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ABSTRACT 

At a 1973 workshop held in Denver^ Colorado, 21 
participants familiar with the uses of Scientific and Technical 
Information (STI) services gathered to identify the system's major 
research needs. Initially, 50 research projects were suggested in the 
areas of design, management, operations, marketing, and global 
considerations. Using a priority weighting technique, the list was 
reduced to the 10 most critical research needs, and five of these 
were selected for discassion about appropriate research designs.\ The 
appendices of this document include the lists of researchable 
questions and the quantitative results from the allocation rating 
forms. (EMH) 
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^ INTRODUCTION 



Interest in the .needs and requirements of users and in increasing the involvement of users in the 
design of science information services has expanded rapidly over the past several years. Major reasons for 
this include the recent and sharp increases in the production, distribution, and cost of scientific and 
technical information (STI). Partly in response to these trends, many individuals and organizations have 
begun to study and to influence the ways that scientists, engineers, managers, citizen activists, and 
others use STI. 

To help identify what user and use problems should be given highest research priority in the near 
future, the National Science Foundation arranged for a workshop In Denver in November 1973 at which 
21 industrial, governmental, and academic representatives generated and set priorities on 50 critical 
research issues. Professor Albert H. Rubenstein of Northwestern University chaired the workshop^.^ 
Participants also completed two post-workshop evaluations of the original set of STI user/use issues. 
Research economist James E. Freeman of the Denver Research Institute coordinated this followup 
activity and preparation of this report. 

We hope, the ideas presented here will stimulate discussion of the issues and how they might be 
studied. Persons with questions or suggestions concerning these issues, or with ideas concerning other 
issues that should be considered, are invited to contact any of us who participated in the workshop. 



Joel D. Goldhar 
Director, User Support Program 
Office of Science Information Service 
National Science Foundation 
Washington, D. C. 20550 
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SUMMARY 



Problem and objectives. Understanding the users and uses of scientific and technical information 
(ST!) has become increasingly difficult during the past decade. At the root of this difficulty, five major 
trends have been operative: (1) an increased emphasis on socially-jelevant research and development 
activity; (2) a concomitant escalation in the production of STI; (3) a proliferation of new STl systems 
and services (STl/SS's); (4) information overload; and (5) sharply increased costs in handling STI. The 
principal objective of the workshop project reported here was to identify a set of critical research needs 
and experimental possibilities related to the users and uses of STI. Results of this project should be of 
value to the entire STI community, including: (1) those responsible for proposing that particular 
research projects be undertaken; (2) those responsible for supporting and initiating new patterns of 
research on STI uses, users, and related services; and (3) those concerned with scientific and technical 
information policy. 

Methodology. The project, under funding from the National Science Foundation's Office of 
Science Information Service, proceeded through three stages. In Stage I, a workshop was convened in 
Denver, Colorado on November 26, 27, and 28, 1973; 21 academic, industrial and government' people 
generated and prioritized a set of approximately 100 "critical research issues and potentially researchable 
questlor*-. " Stage II, which occurred between January and March 1974, involved two post-workshop 
evaluations ot the original set of issues and questions. Stage III, from March through May 1974, 
consisted of an effort by three participants to translate a subset of the critical issues into preliminary 
designs for potential experimental projects. 

Principal results. Participants decided that highest, but not exclusive, priority should be given to 
research projects that treat marketing aspects of STI/SS's. In particular, they recommended that research 
be done: to find more useful ways of categorizing the users of scientific and technical information; to 
determine more effective means of familiarizing potential users with STI/SS's; and to learn how to apply 
other traditional marketing techniques (e.g., assessing share-of-market ?,nd competitive position) to the 
design and control of STI/SS's. Other high priority research issues selected by participants include: 
determining the relevance of STI to major social problem areas (e.g., energy, environment, 
transportation); improving the use of information concerning users in the design and management of 
STI/SS's; identifying new kinds of STI intermederiaries; analyzing the impact of copyright law on 
STI/SS's; and assessing the impact of innovations in computer technology on STI systems and services. 

Three participants-Pauline Atherton, Theodore Melnechuk, and William Paisley-made an 
exploratory attempt tp translate five of the critical research issues into potential research projects. 
Results of their efforts are presented in the body of this report. 
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CHAPTER I. PROBLEM AND APPROACH 

On November 26, 27, and 28, 1973, twenty-one academic, industrial, and government^ people vvith 
special interests in the field of scientific and technical information (STI) participated in a workshop 
conducted by the University of Denver Research Institute under a grant from the National Science 
Foundation's Office of Science Information Service. The topic of the workshop w^s The Users and 
Uses of Scientific and Technical Information; its purpose was to identify and explore critical research 
Issues and experimental possibilities related to this topic. 

Participants qualified on one or more of four counts: (a) involvement in research on STI problems; 
(b) professional responsibility for designing scientific and technical information systems or services; (c) 
management of such systems or services; (d) formulating or implementing public policy related to 
scientific and technical information. Some of the participants fit two or more of these categories. 

PARTICIPANTS 

Pauline Atherton, Professor, School of Library Science, Syracuse University. 

Alok K. Chakrabarti, Research Associate, Department of Industrial Engineering 
and Management Sciences, Northwestern University. 

James E. Freeman, Research Economist, Denver Research Institute, Uni- 
versity of Denver. 

Joel D. Goldhar, Director, User Support Program, Office of Science Infor- 
mation Service, National Science Foundation. 

Thomas A. Grogan, Manager, Market Research, McGraw-Hill Information 
Systems Company. 

Joanne Hartley, Research Economist, Denver Research Institute, University 
of Denver. 

James Kollegger, President, Environmental Information Center, Incorpo- 
rated. 

James P. Kottenstette, Research Economist, Denver Research Institute, 
University of Denver. 

Nan Lin, Professor, Department of Sociology, State University of New 
York at Albany. 

Theodore Melnechuk, Coordinator, Research Interpretation Program, 
Western Behavioral Sciences Institute. 

John Murdock, Head, Energy Information Center, Battel le Memorial 
Institute. 

William Paisley, Professor, Institute for Communication Research, Stanford 
University. 

Donald Pelz, Project Director,. Survey Research Center, Institute for Social 
Research, University of Michigan. 

Martin D. Robbins, Senior Research Economist, Denver Research Institute, 
University of Denver. 

Albert H. Rubenstein, Workshop Chairman, Professor, Department of 

Industrial Engineering and Management Sciences, Northwestern Uni- r 
versity. 
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Theodore W. Schlle, Research Economist, Denver Research Institute, Uni- 
versity of Denver. 

David Staiger, Director, Information Programs, Society for Automotive 
Engineers, Incorporated. 

Clarence Sturdivant, Supervisor, Technical Information, Marathon Oil 
Company. 

Charles Thompson, Associate Professor, Department of Industrial Engi- 
neering and Management Sciences, Northwestern University. 

Peter Urbach, Deputy Director, National Technical Information Service, 
U. S. Department of Commerce. 

Donald VanDyken, Manager, Product Development, Information Handling 
Services, Incorporated. 

The following persfins also contributed ideas to the workshop even though they could not attend 
'the sessions: Wayne Boucher, Secretary, Senior Project Manager, The Futures Group; Walter Carlson, 
Manager, Advanced Marketing Development, International Business Machines Corporation; James May, 
Director, Center for Communication and Information Research, Graduate School of Librarianshjlp, 
University of Denver; Joseph Szyliowicz, Associate Professor, Graduate School of International Studies, 
University of Denver; and John Welles, Head, Industrial Economics Division, Denver Research Institute, 
University of Denver. 



BACKGROUND 

The workshop, in one sense, was a partial response to several recent trends that have forced 
fundamental changes in the management and use of much scientific and technical information. Those 
trends include, but are not limited to, an increased emphasis on socially-relevant research and 
development activity; a concomitant escalation in the production of STI; a proliferation of new STt 
systems and services; information overload; and sharply increased costs of handling STI.* 

An important factor underlying these changes has been the emergence of somewhat different, if 
not entirely new, needs for scientific and technical information systems and services (STI/SS's). What are 
those emergent needs? How do they differ from one audience or user group to the next? How 
effectively and economically are scientific and technical information (STI) systems and services 
responding to emergent requirements? 

That adequate answers to these and related STI questions are yet to be found is reflected in the 
recent and thoroughgoing reorganization of the National Science Foundation's Office of Science 
Information Service. The establishment of the User Support Program was particularly relevant to the 
conduct of this workshop. The new program suggests that nontraditlonal research questions soon must 
be addressed concerning the variety of traditional and nontraditlonal STI users, uses, delivery systems, 
and service concepts. 



*The nature and significance of these trends has been pointed out by the Committee on Scientific and Technical 
Information {Scientific and Technical Communication, Washington, D. C: National Academy of Sciences, 1969); by J. 
E. Freeman, J. P. Kottenstette, and M. R. Bobbins {New In forma tidn Services in Social Problem Areas, Denver: 
University of Denver Research Institute, 1973); and by W. T. Knox ("Systems for Technological Information Transfer," 
Science, 181, 3 August 1973, 415-419. 
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The National Science Foundation supported this workshop to help in (1) reviewing what progress 
has been made to date in identifying and treating STI use issues, especially barriers to effective STI use; 
and (2) providing researchers interested in investigating such issues with an opportunity to develop the 
beginning of a research "agenda" for the field. 



The workshop itself was loosely organized around approximately 50 critical research issues which, 
in varying states of refinement, were formulated in advance by participants.* The first raw list, including 
a number of potentially researchable questions related to many of the issues, is presented in Appendix 
A, 

Workshop participants decided 'ate in the meeting to take part in two post-workshop evaluations 
of key STI issues and researchable questions. The prime purpose of- the first^ post-workshop evaluation 
was to reassess the issues and questions after (a) redundancy in the original list had been reduced, and 
(b) the set of issues had been subgrouped into logical categories. The second evaluation round form is 
reproduced in Appendix B. 

The second post-workshop round of evaluation served still another purpose, that of suggesting to 
potential STI research sponsors which of the issues might receive the highest funding priority in the near 
future. Each participant was asked to assume that a total of 1,000 points could be allocated across the 
thirty STI issues that had received the highest overall ratings in the previous two rounds of evaluation. 
Chapter H presents and analyzes the results of this "1,000 point allocation" exercise. 

Identifying and evaluating the importance of STI use and user issues would be sterile if no attempt 
were made to demonstrate how those issues could be translated into experimental projects. To this end, 
three workshop participants-Atherton, Melnechuk, and Paisley-were asked to formulate a series of 
potential research projects related to a subset of STI issues identified and evaluated by participants as 
areas of critical research need. Their suggestions for possible experiments are presented in Chapter III. 
They are presented only as illustrations of the process of using the issues identified in Chapter II to 
develop research projects. It should be possible for readers of this report to replicate the process for any 
of the other issues that might be of interest. 



What areas of concern relevant to STI uses and users should be considered of critical importance 
during the next three-to-five years? What research methodologies, in the form of field or administrative 
experimentation particularly, should be employed in efforts to facilitate progress in treating those issues? 
Given limited resources, what might be the tradeoffs when certain research initiatives are undertaken 
while others are not? 

The Delphi-like technique employed in meeting the objectives of this project generated potential 
answers to these questions. By inviting participants to evaluate the original list of research issues and 



*The phrase "research issue" connotes an unresolved problem amenable to systematic (often scientific) analysis 
and judgement. A "critical" issue is a problem that may be decisive in the scheme of things. 
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questions, and then to reevaluate subsequent revisions of that list, an appreciable reduction of initial 
divergences occurred. The three rounds of evaluation also permitted quantitative processing of essentially 
qualitative issues. From these deliberations, this report was shaped; hopefully, it will serve as useful 
input to current thinking concerning what research should be conducted into the uses and users of 
scientific and technical information. 
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CHAPTER II. CRITICAL RESEARCH NEEDS 



The process of deciding what scientific and technical information issues are of critical impoi^tance 
is as much a product of history as it is of reason. Reason plays a part, of course, since persons making 
such decisions attempt to weigh the alternatives as effectively as possible. Their decisions, however, are 
limited by their opportunities which, in turn, are limited by history-that is, by existing commitments, 
by the types of institutions currently disseminating STl, and, in a more fundamental way, by the 
perspectives they have concerning the nature of "scientific and technical" information, 

Midway through the workshop, a significant difference was discovered in participants' orientations 
to the STl area. This chapter analyzes the nature of the difference, describes the way the discovery of 
the difference led to the formulation of a new perspective, and, finally presents a list of top-priority 
research needs within the framework of the emergent perspective. 

GLOBAL VERSUS FAMILIAR PERSPECTIVES 

During the small group meetings that were interspersed between plenary sessions, participants 
suggested and reacted to dozens of different ideas. One participant, James Kottenstette, noted that his 
particular group was experiencing "communication difficulties" during the consideration of various STl 
issues and research needs. He further noted that others in his group, and in the other three small groups 
as well, shared his concern over such difficulties. 

After some probing, it was hypothesized that two different perspectives concerning STl systems 
and services were at the core of these difficulties: one perspective was labeled global, the other familiar. 
Participants having a global mew were concerned principally with STl user/use issues that ordinarily lie 
beyond the control of STI/SS designers and managers. Examples of "global" issues include recent 
changes in copyright law and practice; the emergence of new priorities in research and development; and 
the trend toward finding and borrowing, rather than generating, scietitific and technical information. 
STI/SS designers and managers must try to understand and cope with global issues; however, they 
typically do not regulate the forces that created or currently affect such matters. Globally-oriented 
participants tended to evaluate the criticality of issues, not surprisingly, with a set of criteria derived 
from the nature of the contexts or environments within which STl/SS's are operated. 

In sharp contrast, participants with a familiar view of STl/SS's preferred to focus attention on 
user/use issues that characteristically fall under the functional control of STI/SS designers and managers. 
Illustrations of "familiar" issues involve, among other things, the use of information concerning user 
behavior in STI/SS design; the segmentation of STl markets; and networking. STI/SS designers and 
managers are fairly free, within resource limitations, to determine who should do what, how, and when 
to deal with, the problems implied in familiar issues. Participants more interested in familiar issues 
assessed the importance of such issues on the basis of considerations essentially internal to the design 
and operation of STl/SS's. 

Another interesting difference was discovered between "globals" and "familiars" when they were 
asked to define the basic components or elements of STl systems and services. Analysis of their written 
definitions revealed that "globals" tended to use nouns (e.g., media, sources, messages, users), while 
"familiars" more often used verbs and verbal derivatives (e.g., generation, acquisition, retrieving, using). 
The meaning or significance of this difference is not at all clear. One possible interpretation is that the 
language used in defining STl/SS's simply reflects the degree of control designers and managers perceive 
that they exercise over issues they confront: the less control they think they have, the more useful 
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"nounal" definitions become, since they capture the basically passive position of STl/SS's in the context 
of larger "external" forces. Conversely, the more control designers and managers perceive they can exert, 
the more appropriate "verbal" definitions are, because they underline the active ability of designers and 
managers to formulate and regulate STl/SS activity. 

THE EMERGENT "FIVE LEVELS" PERSPECTIVE 

The discovery of the different perspectives or orientations toward STI systems and services became 
a ma.tter of major concern in subsequent considerations of critical STI research needs. Post-workshop 
analysis of the original list of issues, which are presented in Appendix A, revealed a 7-to-1 ratio of 
familiar-to-global issues deserving STI research emphasis in the immediate future. 

To simplify the second and third rounds of. evaluating and prioritizing the issues, a strategic 
decision was made to organize the familiar issues into four "internal" or "functional" levels of concern 
to STl/SS designers and managers; (a) design; (b) management; (c) operations; and (d) marketing. Design 
refers to the planning process that ordinarily precedes the establishment and operation of STl/SS's; this 
term connotes the specification of system objectives as well as selection of the methods, techniques, 
hardware, and other means to be used in accomplishing those objectives. Management, operations, and 
marf<eting concern various interrelated aspects of STl/SS's that follow the design process. Management 
Includes the organization, financing, and evaluation of STi/SS's. Operations involve the procedural 
dimensions (e.g., training programs, publication, distribution) of STl/SS's. Marketing encompasses the 
subset of STl/SS activities related to identification of potential STI users, determining their preferred 
modes of obtaining STI, assessing the competitive posture or various information services, and the like. 

These four functions, depicted in Figure 1, comprise issues over which STI system and service 
managers ordinarily exercise some degree of control. An additional level, (e) context, was added to cover 
global STI issues and questions related to STI uses and users, but typically beyond the control of STI 
system and service managers. 



Figure 1. Five Levels of Concern Related to Uses and Users of Scientific and Technical 
Information Systems and Services (STl/SS's)* 




*Not an attempt at a rigorous taxonomy or model of the STl/SS domain, but merely a convenient way of grouping 
issues and questions related to STi uses and users. 
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EVALUATION CRITERIA 



The ground rules for assessing the importance of STI user/use issues were straightforward, but 
difficult to apply. First, an issue that has a possibility of exercising a practical and significant effect on 
policy, design, or management aspects of STI/SS's should be rated as more important than an issue 
where such a possibility is judged to be remote. 

A second criterion concerned the researchability of questions related to each of the issues. 
(Appendices A and B list the questions for each issue.) An issue for which highly researchable questions 
had been generated should be rated as more important than an issue for which highly researchable 
questions had 'not been formulated. The obvious reason for this criterion is utilitarian: performing 
research requires the statement and treatment of specific questions. A recognized weakness of this 
criterion 's that the ability to generate researchable questions does not, ipso facto, mean that the issue 
with which the questions are associated is important; conversely, an inability to structure researchable 
questions does not suggest that an issue is trivial. The important point of this second ground rule is 
simply that issues for which researchable questions exist can be translated more readily into specific 
projects. 

A third, but only implicit, ground rule concerned the adequacy of previous research related to the 
issue. An issue on which research had been conducted should be rated as more important when previous 
research could be shown to be overly narrow, untimely, methodologically deficient, or somehow 
otherwise inadequate. A major bone of contention during the workshop, one that brings the central 
importance of this third criterion into sharp focus, dealt with the adequacy of previous STI user studies. 
One of the ironies uncovered in the discussion of this topic was the contrast between the large number 
of STI user studies and the general disregard of such studies by designers of many STI systems and 
services. This debate, and others like it, raised questions of basic significance to much future STI 
research; certain of those questions wilF^be treated in some detail in Chapter IV. Suffice it to note here, 
however, that participant judgement of the adequacy of previous research was a nontrivial factor in 
assessing the criticality of issues. ^ 

CRITICAL RESEARCH ISSUES 

Participants generated an initial set of 50 research issues that they judged to be of major 
importance. Using an 11-point scale of 0 (lowest) through 10 (highest), they then rated the relative 
importance of each issue and the researchability of any related questions. The original set of issues and 
questions is presented in Appendix A. 

The reassessment of issues during the second round of evaluation led to a weeding out of 20 issues 
judged to be relatively less important. The rating form used in the second round is included in Appendix B. 

Finally, in the third evalu'ation round, participants reexamined the 30 issues that survived round 
two and collectivc^ly assigned 18,990 points to those issues they iQ\t 6Biir\r€ ihB highiB'st^rio'rit]r''\n terms 
of future STI user/use research. Third round evaluation results are arrayed in Appendix C. 

Quite significantly, ten of the 30 surviving issues were assigned one-half (51.3 percent) of the total 
number of points allocated. Those ten issues are identified in Table 1. 

Analysis of the data in Table 1 and Appendix C suggests that a70 level of concern— design, manage- 
ment, operations, marketing, or context— has been adequately researched to date. Each of these levels re- 
quires significant and prompt research attention. 



TABLE 1. THE TEN ISSUES RATED AS MOST CRITICAL 



Level of 
Concern ' 


Issue 
Identifier* 


Points 
Assigned 


Issue 


Marketing 


D2 


1,280 


Market segmentation 


Marketing 


D1 


1,240 


Other aspects of STl/SS marketing 


Design 


Ag 


1,095 


Relevance of STI to major social problem 
areas 


Context 


E3 


990 


Impact of copyright law and practice on 
STI/SS's 


Operations 


C4 


930 


Data base subsetting in social problem 
areas 


Design 


A3 


915 


Use of information concerning users in 
the design of STl/SS's 


Management 


89 


865 


Evaluation of STl/SS impacts 


Design 


A7 


845 


Other kinds and potential kinds of 
intermediaries besides traditional kinds 


Management 


B1 


805 


Application of economic analysis 
techniques to the STI sector 


Marketing 


D6 


770 


User role in influencing the managment 
of STl/SS's 



*The unique level identifier and number assigned to each issue in preparing the Second Round Evalu- 
ation Form (see Appendix B). 



Not all levels, however, are equally important. Data in Table 2 indicate that, in the opinion of 
participants, the four functional (i.e., ''familiar''), levels of concern sliould be given much higher priority 
than the contextual (i.e., "global") level. Perhaps this is because of the low perceived researchability of 
questions, or the lack of researchable questions, associated with global issues. Then, too, this finding might 
reflect the basically passive character of STI systems and services when confrontedby global issues. 



TABLE 2. OVERALL RAIMKIIMG OF STI USER/USE ISSUES 
BY LEVELS OF CONCERN 



. Level 




Ranking in Thirds 




Total 


of 


Top 


Middle 


Bottom 


No. of 


Concern 


Third 


Third 


Third 


Issues 


Design 


3 


2 


1 


6 


Marketing 


3 


1 


1 


5 


Management 


2 


3 


2 


7 


Operations 


1 


2 




3 


Context 


1 


2 


6 


g 


TOTALS 


10 


10 


10 


30 



V 



8 



U3 



Among the four functional levels, major differences exist In priority concerns. STI marketing, 
operational, and design issues outran f< tfie management issues, as shown in Table 3. This may reflect aVf 
assumption by participants that STI/SS management issues have been somewhat better researched than 
issues in the other three functional levels. 

*\, 

TABLE 3. OVERALL RANKING OF FUNCTIONAL ISSUES 
BY LEVEL OF CONCERN 



Level of 


Average (X) 


Number of 


Number of 


Concern 


No. of Points 


Issues 


Points 


Marketing 


857.0 


5 


4,285 


Operations 


770.0 


3 


2,310 


Design 


753.3 


6 


4,520 


Management 


595.7 


7 


4,170 


TOTALS 


727.9 


21 


15,285 



Finally, it should be noted that one issue, market segmentation, stands out above all the other 
specific issues as the most important area of critical research need. The significance of this issue will be 
explored in Chapter IV. The basic message, however, is clear: totally inadequate conceptions concerning 
STI users are guiding the thinking of many STI/SS designers and managers. Unless and until research is 
done that reshapes current "mental pictures" of who can or might use STI/SS's, hopes for seeing major 
improvements in the delivery and utilization of such systems and services are bound to be frustrated. 

SUMMARY 

In their effort to determine what research is needed on the users and uses of STI, participants first 
generated a tentative list of 50 key issues. The original list of issues was subdivided into five levels of 
concern: four "familiar" (i.e., STI/SS design, management, operations, marketing); and one "global" 
(i.e., context). Participants then reduced the number of issues from 50 to 30 during the second round of 
evaluation. Finally, they selected ten of the surviving 30 issues as the ones most in need of immediate 
research attention. All five levels of concern were represented by the top ten issues; however, familiar 
issues dominated global concerns, and marketing issues far outranked all others. 
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CHAPTER III. POTENTIAL RESEARCH PROJECTS 



The goals of this project included not only generating a preliminary agenda of needed research, 
but also an exploratory testing of the agenda's usefulness. How would anyone interested in developing 
research projects treating STI user/use topics proceed with the list of critical issues as a base? 

Three participants— Pauline Atherton, Theodore Melnechuk, and William Paisley— were asked to 
address this question. The guidelines they followed required simply that their development of potential 
research projects include: (a) the formulation of sped Tic research questions to be addressed; (b) 
generating illustrative propositions that could be tested empirically; (c) citing relevant literature and 
other information sources; (d) suggesting appropriate research methodologies; (e) estimating the cost of 
conducting the research; and (f) assessing the potential utility of the research. 

To provide for broader coverage of the issues, a decision was made not to have duplicate 
development of any single issue. Thus, the three participants treated five different issues as shown in 
Table 4. The breadth in coverage that resulted involved an important tradeoff: any one pf the five issues 
selected could have been developed into potential research projects in many different 'Ways. Resource 
limitations simply did not permit this luxury. What follows, therefore, should be viewed as attempts to 
illustrate the process of using the issues identified in Chapter II and Appendices A, B, and C to\evelop 
possible research projects. It should be possible for readers to replicate the process for other issues of 
interest. 



TABLE 4. PARTICIPANTS' SELECTIOIM OF FIVE CRITICAL ISSUES 
TO ILLUSTRATE RESEARCH POSSIBILITIES 



Participant 
Responsible 



Issues(s) Selected 



Atherton 



• STl/SS User Training (D4)* 

• Relevance of STI to Major Social Problem 
Areas (A9) 



Melnechuk 



• Other Kinds and Potential Kinds of Inter- 
mediaries Besides the Traditional Kinds (A7) 



Paisley 



• Use of Information Concerning Users in 
designing STl/SS's (A3) 

• Use of Information Concerning Users in 
managing STl/SS's (B6) 



*The issue identifier, here D4, assigned to each issue in preparing the 
Second Round Evaluation Form (see Appendix B). 




* 



STI/SS USER TRAINING (D4) - ATHERTON 



Some specific questions. (1) How are scientists, engineers, managers, and otiiers trained to use 
STI/SS's? (2) What kind of training is needed 'to prepare scientists, engineers, and other STI users for 
interesting and successful use of STI/SS's? (3) How much do alternative approaches to STI/SS user 
training cost, who pays, and what are the results? (4) How can current approaches to STI/SS user 
education be improved? 

Illustrative propositions. (1) There are no formal training techniques for scientists, engineers, 
managers, or others in the use of STI/SS's In general, or for particular systems and services which are 
widely used. (2) Specially designed user manuals, on-line tutorials, and seminars are needed to prepare 
scientists, engineers, managers, and others for interesting and successful use of STI/SS's. (3) Educ:;tional 
institutions, professional societies, governmental agencies, and industry can support training costs for 
STI/SS users with measurably beneficial results. (4) Instructional technology and evaluated message 
design can improve current STI/SS user education. 

Relevant literature. W. Lancaster's On-Line Information Retrieval, particularly chapters on user 
instruction. J. Magnino's chapter in Volume 6 of the Annual Review of Information Science and 
Technology, especially pages 223 and 240 on user manuals and terminal assistance. H. Grobman's 
Developmental Curriculum Projects. The UNISIST Handbook For Training Users of STI. User manuals 
from the ^Educational Resources Information Center, the National Aeronautics and Space Administra- 
tion, the National Library of Medicine, • Lockheed, System Development Corporation, and other 
organizations operating STI/SS's. Guides to available training literature from professional and tecfinical 
societies such as the American Chemical Society, the Society of Automotive Engineers, and the 
American Psychological Association. 

Research approach. For Question 1 above, required techniques include survey research, content 
analysis, and field measurement of STI users' skills. Question 2 calls for an experimental design, under 
field conditions, which would test and measure a variety of teaching techniques for different user 
groups. Question 3 requires a cost analysis of the results of research on Question 2,''with a survey of 
possible funding sources and analysis of the effectiveness of alternative teaching methods. Finally, 
Question 4 could be studied by combining analysis of existing educational methods with the results of 
research on Question 3 and consultation with instructional technologists and learning experts. 

First cost approximation. Research on all four questions would, require from three-to-four calendar 
years, and from five-to-six professional years of effort. Total cost would equal approximately $275,000: 
$30,000 for Question 1, $200,000 for Question 2, $20,000 for Question 3, and $25,000 for Question 4. 

Potential utility. Results of the proposed research could help improve the strategy for achieving 
maximum utilization of government-supported STI/SS's. They also could provide guidelines to American 
government and industry on how to design, fund, and program STI/SS user training. Results also would 
be useful in efforts to coordinate industry-wide, national, and international training in the use of 
STI/SS's. Finally, such research would increase the marketability of U. S. instructional technology 
products for training STI/SS users world-wide. , 

RELEVANCE OF STI TO MAJOR SOCIAL PROBLEM AREAS (A9) - ATHERTON 

Some specific questions. (1) How is STI integrated with other classes of data and information 
(e.g., legal, social) in the development of public policy? (2) How can STI generators be influenced to 
become more participative in responding to policy use requirements for STI? (3) To what extent has 
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STl from one field been integrated with ST! from other fields; where and under what conditions? (4) 
What transformations in STl occur as innovations move along the research-development- 
commercialization continuum? (5) What strategies are employed by STI/SS users in extracting 
information relevant to emergent social problem areas? 

Illustrative propositions. (1) The approach taken in atomic energy to integrate STl with other 
classes of data and information would be more useful than approaches take.. !n other traditional areas 
for deciding how to integrate STl with other information in emerging social problem areas like energy 
and environment. (2) Mission-oriented STI/SS's are more useful than discipline-oriented STI/SS's for 
understanding how to integrate STl with other classes of information. (3) Measurable and manageable 
transformations of STl occur as its use moves from research to commercialization and public use. (4) 
STl generators can do more than they have to date in facilitating the integration of STl with other 
classes of information in emerging social problem areas. 

Relevant literature. G. Anderia, Information in 1985. E. Parker's chapter on "Information and 
Society" in Volume 8 of the Annua/ Review of Information Science and Technology. Various reports 
from the Science and Technology Division and the Legislative Reference Service of the Library of 
Congress. M. William's reports on Selective Dissemination of Information services for educational and 
industrial communities. Recent reports of annual meetings held by the Association of Selective 
Dissemination of information Centers. 

Research approach. Question 1 would require field interviews. So, too, would Question 2, coupled 
with case studies. To research Question 3, it would be necessary to survey the literature on the 
development of specialized, cross-disciplinary STI/SS's. Question 4 would require extending the work of 
Stanford University's Institute for Communication Research on writing STl for lay audiences; it would 
involve field testing STl transformations with different audiences in several emerging social problem 
areas. Finally, Question 5 would require field interviews with representative samples of STl generators 
and the managers of STI/SS's. 

First cost approximation. Investigation of all five questions would involve approximately two 
calendar years, and from 3.0—3.5 professional years of work. Total cost would approach $165,000: 
$25,000 for Question 1, $25,000 for Question 2, $15,000 for Question 3, $75,000 for Question 4, and 
$25,000 for Question 5. 

Potential utility. Research on these questions should generate useful guidelines for government 
officials responsible for planning, designing, and operating information services in emergent social 
problem areas. In addition, the research should be useful for improving the integration of STl in areas 
like energy, environment and transportation. Finally, it should facilitate the development of a more 
rational basis for integrating STl across many fields. 

OTHER KINDS AND POTENTIAL KINDS OF INTERMEDIARIES 
BESIDES THE TRADITIONAL KINDS (A7) - MELNECHUK 

This analysis of nontraditional intermediaries between STl originators and users focuses attention 
(1) on the intermediaries'^ involvement with originators of STl as graduate students, authors of grant 
proposals, conductors of ongoing research, conference participants, authors of journal articles, and users 
of STl; (2) cn the intermediaries' involvement with other professional users of STl (e.g., legislators, 
judges, journalists, TV personalities); and (3) on the intermediaries' involvement with lay users of STl 
(e.g., lecture attendees, citizen action groups, patients, children). 
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Originators as graduate students. Proposition A: Graduate studerits of science and engineering, who 
will eventually be the originators of STI, are not all required to take a course on when and how to write 
a journal article and other transforms of STI. Question A: How many faculty members offer explicit 
courses? How many and which graduate schools and departments require students to take such courses? 
Relevant literature: F. P, Woodford (ed.), Scientific Writing for Graduate Students: A Manual on the 
Teaching of Scientific Writing, prepared by the Committee on Graduate Training in Science Writing, 
Council of Biology Editors, New York: Rockefeller University Press, 1968. Research approach: mailed 
questionnaires. 

Proposition B: Graduate students of science and engineering, who will eventually be the originators 
of STI, are not all required to take a course on how and when to write popular articles and other 
popular STI transforms, or on how and when to cooperate with public relations personnel and 
journalists. Question B: How many faculty members offer explicit courses? How many and which 
graduate schools and departments, if any, require students to take such a course? Relevant literature: 
Science News Communication: A Guide for Scientists, Physicians, Public Relations Officers, and 
Information Specialists, Sea Cliff, New York: The National Association of Science Writers, Inc., 1968. 
Research approach: mailed questionnaires. 

Originators as authors of grant proposals. Proposition C: Some funding agencies (e.g., the Social 
and Rehabilitation Service of the U. S. Department of Health, Education, and Welfare) are requiring that 
grant proposals and grant renewal applications include an STI dissemination and utilization plan. 
Question C: Which funding agencies do this, public and foundation? What are their requirements and 
how do they differ? How have these requirements been changing? What kind of compliance have they 
been given? Relevant literature: Special Instructions to Applicants, SRS Form OFIVl-3, Washington, 
D. C: Office of Planning and Research, Social and Rehabilitation Service, U. S. Department of Health, 
Education, and Welfare, 1974. Research approach: mailed questionnaires and telephone interviews. 

Proposition D: A grant proposal is often the best, indeed the only, piece of literature to select a 
worthy problem, review its literature, and suggest a new way to tackle it. Albeit a piece of sales 
literature loo, the grant proposal gives a coherent view of a problem, and those proposals that are 
funded have passed a critical review process. Therefore, a Journal of Winning Proposals in each field, 
which published at least the conceptual core of winning proposals, could be a valuable herald of 
forthcoming research, new concepts, and a model for applicants. Question D: What conditions would 
proposal authors place on publication? Would funding agencies be forced by the laws on access to 
information to cooperate? Would users really buy such an STI transform? Research approach: market 
research. 

Originators as conductors of ongoing research. Proposition E: Directors of large industrial 
laboratories are not always satisfied with the monthly-written progress reports they receive. Question E: 
To what extent do large industrial laboratories employ people as systematic internal observer-reporters? 
What sort of people are good at this work (e.g., drop-out scientists, ex-English majors who like science)? 
Research approach: mailed questionnaires, telephone interviews. 

Proposition F: Academic scientists are differentially willing to be telephoned or written to on 
technical questions, as a duty to colleagues, graduate students, and serious journalists. Question F: With 
what kinds of information would members of professional societies be willing to have their names 
indexed in the society directories (e.g., major problem of current interest, technique most expert at, 
substance or sp.ecies most knowledgeable about, office or laboratory telephone number, days and times 
available)? What use has been made of directories that do index source people? To what extent could a 
national referral service supplement professional society directories and expedite communication between 
researchers and publics other than peers? Relevant literature: R. Lazzell, Points of Contact: Scientific 
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and Technical Information, Washington,. D. C: National Technical Information Service, 1972. Research 
approach: mailed questionnaires. 

Proposition G: Department chairmen and institute and research directors are intermediaries 
between their faculty members, students, and staff members, and creative or learned experts, whom they 
select and invite to give seminars' on ongoing research and unpublished information and ideas. Question 
G: How many such seminars are given in each field? How many of what kinds of persons attend? Would 
newsletters, trade journals, or primary journals publish lists of such speakers-^nd the titles of their talks? 
Relevant literature: monthly lists of Sigma Xi lecturers published in American Scientist. Research 
approach: mailed questionnaires and analysis of departmental calendars. 

Proposition H: The quickest way to get a grasp of the "hot" information in a field is to interview 
the leaders of research in it, who either originate, or get to hear about, most new developments in their 
field. 

Proposition H ': Grapevines can be tapped through structured telephone interviews by personnel at 
infoi-mation and analysis centers who have done their homework. Question H': What does the literature 
of grapevine telephone surveying have to say about the willingness of scientists to be interviewed in this 
way? How can access be gained to information on privileged grapevines? How can grapevines be tapped 
most effectively for state-of-the-art surveys? How can early stages of science be tapped in advance of 
formal publication to obtain new concepts, new approaches, new data, new interpretations, and 
promising newcomers? How can invisible colleges be tapped for wisdom otherwise not evoked? Relevant 
literature: J. Jordan, "Dial G for Grapevine: A Conversation in Exceptional Child Research," in 
Dimensions: Annual Survey of Exceptional Child Research Activities and Issues — 1970. Research 
approac'i: literature search and analysis. 

Proposition H ": "Centers of Survey for Synthesis" (e.g., the Neurosciences Research Program 
founded in 1962 in Boston, Massachusetts) can convene leaders of research on hot topics as 
"sense-making bodies" to select relevant research findings from many disciplines; clarify major concepts, 
schemata, discrepancies, and controversies; state the latest positive and negative findings, and explain the 
logic of their interpretation; state claims explicitly, in summary form, and evaluate their reliability; make 
conceptual models explicit; attempt provisional synthesis, both intra- and interdisciplinarily; identify 
major gaps in knowledge and obstacles to progress; discuss new methodologies and new strategies of 
investigation, and give hunches as to outcomes; predict new developments; explicate implications for 
policy and action; and cite and give references to and annotate selected literature relevant to each of the 
above. At such centers, artistic mind styles— with their quicker tempo of project completion, ease with 
drafts and provisional visions, and impatience with inexplicitness-~can usefully interact with slower-paced 
scientists uncomfortable with premature closure to produce early-stage maps of progress useful to those 
in and out of a field. Question^ H": What was the experience of the Neurosciences Research Program, 
and is it generalizable? What role did it play in legitimizing an entity-based discipline, neuroscience, in 
place of a set of technique-based disciplines? In what ways did it fractionate the preparation of 
workshop reports between scientists as quality controllers and writers and editors as draft generators? To 
what extent and in what ways can artistic mind styles supplement scientific and engineering mind styles 
in facilitating STl generation and communication? Relevant literature: F. Schmitt and T. Melnechuk 
(eds.), Neurosciences Research Symposium Summaries, Volumes 1-7, Cambridge, Mass.: MIT Press, 1966 
to 1974; G. Quarton, T. Melnechuk, and F. Schmitt (eds.). The Neurosciences- A Study Program, New 
York: Rockefeller University Press, 1967. Research approach: historical study. 

Originators as conference participants. Proposition I: A set of people exists who ate experienced in 
planning, running, transcribing, reporting, and editing the proceedings of conferences. Question I: What 
unpublished hard-earned wisdom do they have about the mechanics of conferences? Are any stenotypists 
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especially trained for specific scientific fields? Is there a compendium of knowledge about the mechanics 
of various kinds of conferences? Relevant literature: F. Crick, "On running a summer school/' Nature 
(1968) 220 (December 28): 1275-1276; M. Mead, "Conference behavior," Co/tym/b/a University Forum 
(1967) 10 (2, Summer): 15—19; L. Kubie, "Problems of multidisciplinary conferences, research teams, 
and \ourna\s,"^ Perspectives in Biology and Medicine (1970) Spring; 405—427. Research approach: 
literature search, study of programs of annual meetings and invited symposia, mailed questionnaires, 
and interviews with workshop chairmen, working staff, and hired staff. 

Originators as authors of primary journal articles. Proposition J: A Journal of Advance Abstracts 
in each field could publish the abstracts of articles already accepted for publication in cooperating 
primary journals, but not yet published in them. This would give readers a preview of the forthcoming 
contributions to the literature of that field. It also would allow cooperating journals to receive preprint, 
rather than reprint, orders for increased economy of printing. Question J: What has been the experience 
of the first such journal, now known as Abstracts and Reviews in Behavioral Biology? What are the 
opinions of the editors of cocTperating journals and of its subscribers? Relevant literature: Council on 
Biological Sciences Information, Information Handling in the Life Sciences, Washington, D. C: Division 
of Biology and /Agriculture, National Research Council, 1970. Research approach: mailed questionnaires 
and personal interviews. 

Proposition K: Journal editors and the editors of conference programs can be persuaded to require 
that authors provide not only an abstract and key terms, but also a title that is a single sentence stating 
the main claim in their papers. Question K: How were journal editors persuaded to require abstracts 
and, later, key terms from authors? To what extent do incumbent editors already believe in the 
quantum-jump increase in informativeness such titles bring to the literature? How willing are authors to 
provide such titles? Can appended "major claim" statements be made a requirement of publication? Who 
should tag papers with the major claim statements? Relevant literature: J. Bernard and C. Shilling, 
Accuracy of Titles in Describing Content of Biological Sciences Articles, Washington, D. C: American 
Institute of Biological Sciences, 1963; Center for Research in Communication, The Role of the National 
Meeting in Scientific and Technical Communication, Baltimore: Johns Hopkins University, 1970, 
particularly page 34. 

Proposition K ': Councils of Editors in each field of science could play a major role in requiring 
authors to provide title sentences, and in effecting other improvements and economics. Question K': 
How transferrable is the experience of the Council of Biology Editors in this area? What are the 
attitudes of incumbent editors in other fields to this idea? Could NSF-sponsored Councils of Journal 
Editors in each field require authors to provide the main claim? Relevant literature: Committee on Form 
and Style of the Council of Biology Editors, CBE Style Manual, 3rd ed., Washington, D. C: American 
Institute of Biological Sciences, 1972; R. Day et al (eds.). Economics of Scientific Publications, 
Washington, D. C: Council of Biology Editors, 1973; J. Woodford (ed^). Scientific Writing for Graduate 
Students: A Manual on the Teaching of Scientific Writing, i\lew York: Rockefeller University Press, 
1968. Research approach: historical study, mailed questior^naires, telephone interviews, literature 
searches. \ 

Proposition L: The widespread communication of new concepts and new data stemming from new 
discoveries could be accelerated if the coining of new terms (e.g., maser, laser) associated with such 
discoveries were systematically sought for and the new terms periodically published. Question L: How 
many new terms enter the thesauri produced annually by- each major indexing organization? What new 
terms have recently been unknown to translators of science literature in the major world science 
languages? What has been the experience of the American Institute of Physics in preparing its series of 
^ glossaries of terms used frequently nn various hot branches of physics? Relevant literature: List of new 

terms in Index Medicus; possibly the publications of the American Translators Association; also the 
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series of glossaries of terms used frequently in various physics subfields, published by the American 
Institute of Physics since 1962. Research approach: Wterature reviews and telephone interviews. 

Proposition M: The nation's indexing services rest on the labors of a set of people, the indexers of 
the literature, who are a precious resource, especially when papers still lack titles that are maximally 
informative: More and better indexers are needed. Question M: Have indexers been surveyed to learn 
what they think of their education and training, and the practices of their organizations? Could a 
compendium of their wisdom be prepared for the quidance of library schools and the employers of 
indexers? Research approach: mailed questionnaires, personal interviews, and literature searches. 

Originators as users of STI. Proposition IM: Graduate students of science and engineering, who 
eventually will be heavy STI users, are not all required to take an up-to-date course on how to use 
nationally available STI/SS's. Question N: How many faculty members offer explicit courses? How many 
and which graduate schools and departments require their students to take such courses? How are 
practicing scientists, engineers, managers, and others trained to use STI/SS's? Research approach: mailed 
questionnaires. 

Intermediaries relating to other professional STI users. Proposition O: Congress, and the various 
state legislatures, pass and revise laws without the legal requirement that all law be consistent with the 
state of science at the time of passage and be reviewed for such consistency at periodic intervals. 
Question O: To what extent, and in what ways, do any legislatures have a requirement for law to agree 
with science? What effect on law has the Salk Institute Program on Biology in Human Affairs had (e.g., 
the revision recently of Massachusetts laws on drug use and abuse)? Research approach: personal 
interviews. 

Proposition P: Judges make decisions and pass sentences according to their beliefs about behavior 
as well as their interpretation of the law. Some beliefs are, and others are not, brought up to date with 
behavioral and biological science. Question P: To what extent do legal journals attempt systematically to 
interpret new science for judges? Would judges welcome such a service? How do judges think such an 
updating process could best be fashioned? What has been the experience of state medical associations 
with updating physicians? Relevant literature: A new newsletter on social action and the law, Brooklyn, 
New York: Brooklyn College Center for Responsive Psychology. 

Proposition Q: Many journalists who help shape public awareness and understanding of science and 
engineering appear to be unhappy with their experiences and the stereotypes that have come into 
widespread belief. Question Q: How many scientists regret having talked to the press? Why? How many 
journalists resent the treatment accorded them by scientists? Why? What structural and personality 
factors contribute to their regret? Research approach: mailed questionnaires, personal interviews. 

Proposition R: Most of the public gets its STI from TV news programs and talk shows; thus, TV 
commentators, talk show stars, and their guests carry a heavy responsibility for the proper selection of 
subjects and interpretations. Question R: What principles guide the selection of topics and guests? Which 
scientists were guests on the major talk shows in 1973? Which books were plugged? What topics were 
mentioned by Cronkheit, Reasoner, and Chancellor? How many and what kinds of special programs 
were devoted to science and engineering topics during 1973? Relevant literature: program logs and 
scripts. Research approach: mailed questionnaires, personal interviews, analysis of program logs and 
scripts. 

Intermediaries relating to lay users. Proposition S: Popular lectures on scientific subjects are 
marketed by lecture agencies; it is a flourishing business. Question S: What subjects were treated in 
popular lectures during 1973? By whom? What is the academic reputation of the lecturers? Research 
approach: mailed questionnaires, personal interviews, analysis of records. 



16 



Proposition T: Citizen action groups exist on some issues that require the use of scientific 
evidence; often scientists are involved in such groups. Question T: Do such groups tend to be started by 
concerned scientists, or by concerned nonscientists? In what ways do they communicate STI to people 
with political power or civic influence? Relevant literature: T. Vonier and R. Scribner, Community 
Information Expositions: Issue-Oriented Displays and Popular Understanding of Social Problems, 
Washington, D. C: American Association for the Advancement of Science, 1973. Research approach: 
grapevine survey. 

Proposition U: Surgeons require informed consent for operations from prospective patients to 
preclude malpractice suits and other forensic litigation. Question U: What constitutes "informed 
consent" in the 50 states? What printed materials are being developed to do the necessary "informing" 
and to record the "consent"? Relevant literature: New booklets for prospective patients published by 
Health Communication, Inc., Suite 434, Union Bank Building, San Diego, California. Research approach: 
survey of surgeons, state medical associations, and literature search. 

Proposition V: Science books for children are published in large numbers, and TV children's shows 
often have episodes involving scientific claims. Question V: How many titles were published in 1973? 
On what? By whom? How were they reviewed before publication? After? Do any professional societies, 
some of which cooperate in producing textbooks, systematically assume the responsibility for issuing 
public evaluations of these books? Relevant literature: P. Morison's annual review at Christmas time of 
children's science books in Scientific American, Research approach: mailed questionnaires and telephone 
interviews. 

USE OF IIMFORMATIOIM COIMCERIMIiMG 
USERS IN DESIGNING STI/SS's (A3) - PAISLEY 

Some specific questions. (1) Viewing the STI/SS user as an individual "information processing 
system" with perceptual, cognitive, and affective subsystems, what are the major parameters of the 
perceptual subsystem that should affect the design of high-speed or high-density STI interfaces, such as 
proactive displays (via television-like tubes or improved computer-driven screen) that are intended to 
provide the user with a rapid purview of the structure and content of available document files, people 
files, and data bases? (2) According to various attributes of users (e.g., occupation, discipline), what are 
major parameters or factors of the cognitive subsystem that should affect the design of document files, 
people files, and data bases? (3) According to various attributes of users, what are the major parameters 
or factors of the affective subsystem that should affect the design of STI/SS's that maximize positive 
reinforcement and minimize negative reinforcement for users? (4) Across a sample of 
information-seeking sequences or episodes, what are the critical junctures at which the perceptual/ 
cognitive/affective subsystems, either singly or interactively, are stressed or contravened by STl/SS 
hardware or software in such a way that the information search takes a false turn or is terminated 
prematurely? 

Illustrative proopositions. (1) With high-speed, high-resolution, self-pacing displays and hier- 
archically structured or otherwise "chunked,'' . files, the user's perceptual subsystem will prove to be 
capable of processing certain kinds of STI content at the rate of thousands of bits per minute. Kinds of 
content amenable to such presentations will include file overviews, characterization of average and 
outlying file elements, lists of terms that indicate the entrypoints of the file, and standardized graphic 
profiles that can serve as the distinctive "fingerprints" of different files. (2) When STI files are 
restructured to complement the individual cognitive structures and cognitive styles of users, the user's 
knowledge gain and actual use of information will both increase. (3) When STI interactions with users 
are designed to include token reinforcements (e.g., indications of progress toward the retrieval of specific 
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information), users will turn to STI/SS's more often as well as earlier in each information-requiring task. 
In this condition users will also persevere through more steps per search. As a corollary, information 
retrieved in this condition will be judged more relevant than information retrived from an identical 
STI/SS that lacks a planned "reinforcement schedule." (4) By shifting from a reactive to a proactive 
posture (i.e., by pacing and guiding the user) and by incorporating a "tenacity factor" in an STI/SS's 
response to all kinds of user demands (e.g., well-conceived and ill-conceived), the proportion of false 
turns and premature terminations of searches will be reduced, and as a consequence the adjudged 
relevance of retrieved information will be greater, relative to an STI system that provides access to the 
same files in a nonproactive, nontenacious manner. 

Relevant literature. Two distinctive literatures converge in this research area. One is the 
psychological literature of perception, cognition, and affect. The other is the information science 
literature on the' structure and content of files, their surrogatibn systems, interfaces, and use contexts. 
The psychological literature is best accessed through appropriate chapters in the Annual Review of 
Psychology, in addition to occasional benchmark compilations such as the Berelson and Steiner 
propositional inventory entitled Human Behavior: An Inventory of Scientific Findings. The information 
science literature is best accessed through appropriate chapters in the Annual Review of Information 
Science and Technology. No benchmark compilation is known to exist for these topics within 
information science. 

Research approach. In Question 1, the perceptual subsystem would be studied via experiments 
involving alternative presentations of information on a high-speed display. Much generalizable 
information would be obtained from experiments varying the rate of presentation, the stimulus array, 
and the presentation procedure (e.g., scrolling versus total screen blanking and replacement). 
Experiments would be reminiscent of tachistoscopic studies of perception in the purely psychological 
tradition. Concepts guiding the experiments would be derived from the Gestalt theory of perception and 
from perceptual'Capacity research. 

Existing instruments for the study of cognitive structure and cognitive style would need to be 
adapted to STI content areas in studying Question 2; they could then be used for approximate 
classification of STI users into different cognitive structure/style categories. Transforms would be 
developed to restructure a medium-size file in ways that complement these different structures and 
styles. Experiments would present alternative file excerpts to groups of users in counter-balanced design, 
against the twin criteria of knowledge gain and actual use of information. 

For Question 3, a single, relatively deep STI system would be adapted to several conditions of user 
reinforcement. Experiments would test frequency of use and perseverance of use across the several 
conditions, as well as the corollary criterion of retrieval effectiveness (i.e., relevance). 

Finally, case studies of STI users in natural contexts would serve analysis of Question 4 by helping 
Identify critical junctures in information-seeking sequences or episodes at which searches take false turns 
or are terminated prematurely. 

First cost approximation. Investigating Question 1 would require about $50,000 to set up a 
hardware/software configuration that would be flexible enough for a full range of perceptual subsystem 
experiments. About $10,000 would be needed to cover the experiments themselves. Total: about 
$60,000. 

For Question 2, about $25,000 would be required for software and multiple transforms of a 
medium-size file. Another $20,000 would be needed for the extensive subject measurement and data 
analysis that cognitive structure/style experiments require. Total: about $45,000. 
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Research into Question 3 would cost about $25,000 for software, unless combined with work on 
Question 2 for some reduction of total effort. Another $5,000 would be needed for the relatively simple 
measures of use frequency, perseverance, and retrieval effectiveness. Total:- about $30,000. 

About $100 per user would be needed under Question 4 to collect case-study data on critical 
junctures in information-seeking sequences or episodes. Cost of the study would depend on the sample 
size. Assuming that a stratified sample of 250 users would encompass major attributes that differentiate 
user groups, total cost would be $25,000 plus $5,000 study prepraration and data analysis or $30,000. 

Potential utility. The major policy step of shifting STI/SS's from a reactive to a proactive posture 
depends on user data of these kinds. Present systems that are little-used or used with poor effect can be 
shown to stress or contravene users' perceptual, cognitive, and affective subsystems. New systems that 
are designed without user data will also fail in their missions. 

USE OF INFORMATIOIM COIMCERIMING 
USERS IN MANAGING STI/SS's (B6) - PAISLEY 

Some specific questions. (1) What aspects of STl/SS management should be regarded as 
particularly interdependent with use patterns, relative to other aspects that can be regarded as 
engineering or business issues without reference to a particular user population? (2) How should patterns 
of user need or preference optimally guide a fixed-capital-commitment aspect like the development and 
maintenance of document files, people files, or data bases? (3) How should patterns of user need or 
preference optimally guide the variable-capital-commitment aspects of an STl/SS, like referral and 
question-answering services, repackaging and fresh publication, document duplication services, etc.? (4) 
What are the most reliable, valid, and useful ways to gather information about user need or preference, 
according to the aspect of STl/SS management being affected by the data? 

Illustrative propositions. (1) Users regard STI/SS's as interfaces to information. Aspects of STl/SS 
management are arrayed along a continuum of concern/indifference for each user. Of greatest concern 
on this continuum are the content areas covered by the STl/SS, together with corollary aspects of 
comprehensiveness and timeliness. Of next greatest concern are the occasions and formats of service. Of 
less concern than either content or format is service cost, as long as it lies within one or two standard 
deviations of the average STl/SS cost in the user's field. (That is, within the plus-minus 2z range, 
expensiveness will not lead to underutilization nor cheapness to overutilization.) At the indifferent end 
of the continuum lies a set of STl/SS management issues that users neither know nor care about. Across 
a sample of the user population, the amount of concern shown for management aspects can be shown to 
stabilize, with the consequence that STl/SS managers can provide themselves with a rank-ordered list of 
aspects that users are concerned about. The list advises managers to: (a) condition some decisions on 
user data; (b) proceed on other decisions without user data. 

(2) Since decisions affecting fixed-capital-commitments !ike document files, people files, and data 
bases are related to such main trends of user behavior as number of queries, number of documents 
ordered, number of subscriptions placed, etc., clear guidance for acquisition, weeding, and peaking of 
files comes from longitudinal data provided by users in the course of conducting searches, ordering 
documents, placing subscriptions, etc. ror example, acquisition priorities can be rank-ordered by file 
classification area according to users' prevailing demand for each area, while weeding decisions can be 
based on "half-life" statistics rather than subjective judgement about the value of file elements. Other 
decisions, ranging from file description to arrangement of materials for open access, can be based on 
conditional probabilities that can be shown to exist in users' joint use of file elements, rather than on a 
priori file dimensions or relationships that users don't care about. Use of the STl/SS will increase to the 
extent that its management reflects these main trends of user need and preference. 
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(3) Decisions affecting variable-capital-commitment are often unrelated to main trends of user 
behavior and require a different focus for collecting user data. That is, decisions to provide short-term, 
optional, or peripheral services cannot be conditioned by main trends of user behavior~-such services do 
not create data series.. Instead, it is the deviant trend in user behavior that often points to an 
opportunity to make a variable-capital-commitrnent with good effect. When a mechanism exists for 
capturing minor and deviant trends in user behavior, variable-capital-commitment decisions based on 
such trends have a greater probability of success than decisions based only on managerial experience. 

(4) Mechanisms for collecting user data range from volume counts and observations of use to 
questionnaires and interviews that raise a user's perspective above an individual episode of STI/SS use. It 
will be found that vo» me counts and use observations are best adapted to decisions affecting 
fixed-capital-commitments, while questionnaires and interviews are best adapted to decisions affecting 
variable-capital-commitments. The power of questionnaire/interview data is such, however, that all 
management decisions that have a substantial user-input (see PI) will benefit from the collection of 
questionnaire and interview data from small samples of users, conjoint with volume counts and use 
observations based on the entire user population. 

Relevant literature. Since this research area is a conceptual and procedural sequel to the use of 
information concerning users in the design of scientific and technical information systems and services 
(see A3 above), it draws upon the same literature resources. 

Research approach. For Question 1, decisions affecting STl/SS management can be abstracted 
from composite management plans of several STl/SS's. Such lists of decisions can be reviewed by panels 
of users, whose task is to indicate their concern about the outcome of each decision. To create a range 
of hypothetical outcomes for each decision, a question format like the following can be developed: "In 
thinking about the consequences of a decision regarding X, how much would it matter to yoi3[ if.^ 
happened rather than B? If A happened rather than C? If B happened rather than C?" Responses 
indicating relative concern would be summarized to provide a rank-ordered list of STI/SS management 
aspects according to user concern. 

The records of well-managed STl/SS's provide many data series that should condition fixed-capital- 
commitment decisions, the specific concern in Question 2. Additional data series result from analyses of 
user activity that are not normally recorded but easily can be (e.g., number of orders per document title 
per year). Finally, a small amount of questionnaire and interview data should be collected to assure that 
trends in existing data series are not being misread. 

Under Question 3, decisions affecting variable-capital-commitments should be guided by data that 
encompass a set of options at least as broad as the set that the STI/SS manager must realistically 
consider. The research approach needed in this case must in some sense upset users' expectations about 
the usual services of the STl system and cause them to speculate freely about desirable short-term, 
optional, or peripheral services. Generally speaking, an interview is the most effective data collection 
procedure to induce respondents to speculate beyond normal conventions and expectations. The sample 
to be interviewed in this case should be stratified by irr^portant attributes of users as reflected in their 
use of STI/SS's. 

Finally, Question 4 speaks to the need for statistical indicators of STl use that extend both across 
fields and across time for the sake of comparative data. A multiple-operationlst paradigm is appropriate; 
that is, several candidate indicators should be tried for each variable (e.g., user satisfaction) and the joint 
data should be tested for convergent-discriminant validity. Different measurement approaches will prove 
to inform different variables effectively. Superior measurement^;d'pproraches can be arrayed according to 
the aspect of STl management that is at issue. 



20 



First cost approximation. To investigate Question 1, about $10,000 would be needed to set ujD 
and conduct a study on the most effective role for user input in various aspects of STI management in a 
given discipline or field of research. Another $10,000 would be required to replicate the study in two 
additional, maximally contrasting disciplines or fields. 

For Question 2, each STI/SS can maintain data series of main trends in user behavior with very 
little increment in operating costs. Counts and observations can be tallied into useful summaries for an 
average of $.10 per user transaction. Assuming a hypothetical STI/SS that has 100,000 user transactions 
per year, tallying and preparing annual summaries of the main trends of user behavior would cost about 
$10,000 (almost all of which is labor cost distributed across the year at times of user transaction,, 
including the tallying of an average of 400 user transactions per working day). 

Regardless of the size of the STI user population, a sample of N users is appropriate for Question 
3 according to subsample variances. A hypothetical STI/SS that has a user population of 5,000 may 
require a sample of 500 users If it consists of heterogeneous subgroups, or it may require a sample of 
only 200 if there is a considerable homogeneity across subgroups. Based on the need to collect data 
from 500 users, interviews would cost about $10,000, and overall study costs would be about $15,000. 
From one such study, however, an STI/SS could obtain Input that leads to a number of 
varlable-capital-commitment decisions. 

The methodological aspect of Question 4 is extremely hard to cost, because increased budget 
would lead to useful inclusion of diverse measures. To be worth doing at all, however, a statistical 
indicator study of this kind would have to include populations and measures that run the cost to at 
least $50,000. 

Potential utility. At present there is no satisfactory method for incorporating user data into 
STI/SS management processes except within the simplest marketing paradigm of "produce what sells." 
There is no explication of STI/SS management aspects to differentiate those in which user input is 
crucial and those in which user input is unimportant. Within the next few years, the greatest need in 
this research area is for simple, packageable, user-monitoring procedures that can be adopted by 
average-size STI/SS's, costed to overhead, and drawn upon in a number of management processes. 
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CHAPTER IV. CONCLUDING COMMENTS 



This project was undertaken for a very pragmatic reason: to Identify those issues upon which there 
appears to be agreement among specialists in a number of STl-related fields regarding the most 
important problems confronting users of scientific and technical information. Analysis of project results 
presented in Chapters 11 and lil reveals some progress in this direction, at least among workshop 
participants. As interested nonparticipants review these results, it is hoped they will find that a number 
of their own biases about needed research are reflected in the preliminary agenda that emprgcd. 

The principal purposes of this final chapter are to state the intent of the emergent agenda, and to 
make some comments on the limitations of previous research on STI users. 

THE EMERGENT AGENDA 

The emergent agenda obviously is not the only appropriate, nor the "best," one for specifying 
what research is needed on the users and uses of STI. A number of alternative agendas could be 
formulated. They undoubtedly would differ in emphasis, form, and content from the "familiar versus 
global" perspective elaborated here. So why bother with this agenda? 

First, because it is reasonably comprehensive, possesses internal consistency, and is fairly easy to 
use. A second reason for using the agenda is mechanical: it provides a kind of checklist, one that 
increases confidence that important problems are not being ignored. Finally, the agenda can provide 
colleague support for people undertaking research projects in the STI field. 

As reasonable as these considerations may be in themselves,' they are limited and transitory in 
comparison to the original purpose for generating the agenda,- namely, working iteratively to identify 
some of the areas that should be investigated in this field. 

PARTICIPANTS' DOMINANT CONCERN: RESTRUCTURING STI USER STUDIES 

In their analysis of various issues, participants spent the greatest amount of time on two closely 
related matters: the limitations of previous STI user studies, and the need for developing new ways of 
classifying STI users. 

The barefoot cobbler's children. "Can we explore the question of why the cobbler's children are 
running around barefoot?" With this metaphorical question, Urbach directly confronted the frustrations 
many participants had experienced with previous STI user studies. He continued: 

If all these studies we are doing are so good, if all this information we have to convey 
is so good, why don't people in the information business who are hungry for 
information about the user community use any of the studies . . ? 

Three types of user studies. Responses to Urbach's question, though entirely qualitative, were 
quite provocative. Paisley and Melnechuk distinguished three types of "STI user studies": descriptive, 
predictive, and interventive. According to Paisley, the vast majority of previous user studies fall into the 
descriptive category. Results of such studies, he maintained, are necessarily disappointing to STI/SS 
managers who typically work in a predictive mode. Workshop chairman Rubenstein elaborated Paisley's 
analysis by pointing out that Jcscriptive user studies mostly have been conducted by "academics" and, 



predictably, have little or nothing to do with selling information products (i.e., STI systems and 
services). 

Participants did not ignore the fact that a considerable number of predictive user studies have been 
conducted, particularly by commercial information firms, but that the results generally have not been 
available to academic researchers. Bobbins noted that most such studies have been paid for by 
commercial information services that use the results for competitive purposes. 

Few user studies involve interventior^ by the researcher. Melnechuk and Staiger both explored this 
point. In Melnechuk's view, the great challenge confronting STI researchers-and the managers for whom 
they work— is to invent information services and products that people don't even know they want. 
Invention^ agreed Staiger, is the critical first step; the second is to risk the invention in a real situation. 
By "a real situation," Staiger meant one in which dollars, reputations, and ongoing systems are at stake. 

The researcher's stakes in user studies. Staiger intimated that far too few researchers ever become 
involved in interventive studies. He went on: "How many people who have conducted a user study have 
profit and loss responsibility for a product?" 

Staiger's challenge brought a strong rebuttal from Lin: 

1 consider that a very impractical demand ... We certainly cannot expect researchers to 
be also responsible for any specific application of their (information) product... If 
yBa" really wanted to have the researchers to be responsible for this kind of research, 
then what you are asking for is to have thousands of user studies for each specific 
purpose, which is a waste of energy. 

Market research methods needed. It became clear that many academically-based STI user studies 
have been conducted by investigators with similar, usually discipline-oriented, perceptions of users. It 
was at this point that the discussion finally moved from a general analysis of the problem to the 
formulation of a possible research question, one that came to be of dominant concern. Assuming it 
would be desirable to increase the number and quality of predictive and interventive studies, Goldhar 
asked: "/Von/ do \A/e get traditional, existing, well-proven consumer product market research techniques 
applied in the science and technology information business?^' 



A BRIEF EDITORIAL 



This report obviously does not contain answers to the many important and puzzling issues and 
questions that arose during and after the workshop discussions. The agenda of items identified, however, 
should help provide guidelines for further research into the relationships between STI users and the 
systems and services that are designed and provided to serve those users. 

The total amount of money and effort spent on studies of STI user behavior has been quite 
modest compared with the amounts spent for the main scientific activities (e.g., R&D) that STl/SS's are 
supposed to support. If the evaluations of the participants are valid, however, much previous STI user 
research appears to have missed the mark in terms of (1) improving understanding of the phenomena, 
anti (2) improving the design and management of STl/SS's. 

One reason for the apparent inefficiency to date in STI research may be due to the lack of 
conceptual frameworks or theories within which different studies can be related. Approximately 100 
empirical studies of STI user behavior are reviewed each year in Carlos Cuadra's Annual Review of 
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Information Science and Technology; it appears to be very difficult for the reviewers to reconcile some 
of the major results and to relate them to any systematic understanding of the underlying phenomena. 



Another reason for the noncumulative nature and the inconclusiveness of much of this research might be 
the methodologies used to collect the data and structure the studies. There seems to be very little 
systematic replication of specific studies. If such is the case, it is not surprising that the validity and 
significance of much research in the field can be questioned. 

One of the hoped-for results of the workshop and this report is a change in the kind of research 
done in this important area, including the methods used for doing that research. We would like to see 
future research fo cussed on problems that are botbrscientifically researchable {that is, the methods used 
yield highly credible, communicable, and replicable results), and practically important {that is, they can 
be of direct use to people involved in designing, providing, managing, and using STI systems and 
services). Specifically, we hope the results of the workshop; as represented in this report, will be of 
some use to- four different audiences in the following ways: 

To researchers of STI user behavior. Many of the questions raised by participants present 
considerable intellectual challenge. The issues hopefully will stimulate STI researchers to continue or to 
initiate research in this field for scientific as well as practical reasons; to -devote time and effort to 
replication of existing studies so as to improve the credibility and usability of studies that have shown 
promising, but not conclusive results; and to provide a relatively simple framework and context so that 
individual studies can be related to the overall STI process and other research in progress, completed in 
the past, or to be undertaken in the future. 

To designers of STi/SS's. The workshop discussions suggested that very little direct use is made of 
the research results that are available and that appear credible in the design of new and improved systems 
and services for STI users. We hope that at a minimum this report will serve STI/SS designers by pointing 
up some of the key issues in the field and some of the needs their potential clients have and are concerned 
about. We also hope that this report can stimulate a dialogue, where none now exists, between system 
designers and their clients: both the managers of,STI/SS's and the ultimate users {i.e., the scientists, 
engineers, and others engaged in R&D and related activities). We also hope that this report will stimulate 
some of the designers of such systems to undertake or expand actual R&D on user behavior aspects of their 
systems, and to thereby build up a capability for continuous improvement in our knowledge about user 
needs and behavior as well as a large data base for the design, evaluation, and management of STi/SS's. 

To managers of STi/SS's. We hope that this report will stimulate STI/SS managers to carry out more 
in-house research on the user aspects of their STI activities both before they purchase new or improved 
STi/SS's, and after they receive and install them. For example, we hope to see a significant increase in 
"administrative experimentation" by STl/SS managers interested in improving system effectiveness and 
utilization. {STi/SS managers not familiar with, but seriously interested in, administrative experimentation 
could benefit from reading workshop participant Thompson's articles on this subject in the May-June 1974 
issue of Human Factors and in the May 1974 issue of IEEE Transactions on Engineering Management) 

To ultimate users of STi/SS's. In the scientific spirit of intellectual curiosity, as well as for practical 
considerations, we hope that at least a few ultimate users will feel inspired to examine some of these issues, 
let us know their views on them, carry out some experiments on their own, and perhaps receive some 
consolation in knowing that their problems are not unique and that help.may be on the way. 
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APPENDIX A r 

KEY STI ISSUES AND POTENTIALLY 
RESEARCHABLE QUESTIONS: 
Form Used in Evaluation Round One 

1-1. Need for techniques of economic analysis of STI sector. (PA)* 

Q-4. What kinds of economic analysis are potentially appropriate to STI? (CT) 
Q-5. Which types are most useful- for STI? (CT) 

Q-42. Can the effect of a publication be measured by the degree to which it reduces the distance 
between the "current awareness" of a field held by its leaders and that held by its normals? 

1-2. Need for economic models of communication of information. (PA). 

Q-6. How can the STI sector be modeled effectively for economists and managers? (JG, TM, 
PA) 

Q-8. How can the value of information per se be measured at various levels? (DS) 

Q-13. What are the cost/benefits of various STI/SS's? (AR) 

Q-39. How can the efficiency of doing STI operations be improved? 

\ 

1-3. Inclusion of proprietary Information/data in STI. (DS) 

1-4. Interservice pricing. (JG) 

1-5. Structure of STI/SS sector. (JG) 

Q-1. What are the incentives for influencing structure? (JG) 

1-6. Content quality control. (DS) 

Q-2. What quality control mecSanisms are feasible? (JG) 

Q-3. What kind of quality control mechanisms are desireable? (JG) 

Q-7. How can the value of information per Scf be measured atWarious levels? (DS) 

1-7. Nontechnical STI users. (JM) 

Q-1 6. How do public interest groups acquire STI/SS's? (JKg) 

1-8. STI systems and services for public interest groups. (JKg) 

Q-9. How will public interest groups respond to various types of STI/SS's and STI interpreta- 
tions? (JKg, AR) 

Q-10. How will public interest groups stimulate various types of STI/SS's and STI interpreta- 
tions? (JKg, AR) 
Q-1 6. How do public interest groups acquire STI/SS's? (JKg) 

1-9. Role of community libraries in STI. (JKg) 

1-10. User role in influencing the design and management of STI/SS's. (CT) 

1-11. Use of information concerning users in the design and management of STI/SS's. (CT) 

1-12. Integration, standardization, and tailoring of STI data banks (vertical vs. horizontal) for internal 
use in industry. (CS) 

Q-17. What alternatives exist for standardizing the structure of STI/SS's? (JF) 
•Initials of contributors are in parentfieses (see participant list in Cfiapter I). 
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APPENDIX A (Continued) 

1-13. Market segmentation. (JG) 

Q-11. Who are the user groups and what are the bases for categorizing them? (TS) 
Q-18. What different ways can "market segmentation" for STI users be done? 
Q-19. What kinds of tertiary services are needed by different market segments? 
Q-29. What are (the information) purchasing behavior patterns of STI users? 
Q-40. By what principles are users segmented? 

Q-41. Within segments, what are the individual behavior patterns? What are the constraints? What 
systems meet these needs? 

1-14. Choice of target areas for user studies. (TG) 

1-15. Availability of proprietary user behavior information to STI academic researchers. (AR) 

1-16. Inputs of users to user studies. (AR) 

Q-12. What methods would be appropriate for permitting users of user studies to feed into and 
evaluate such studies? (TG) 

1-17. Should the federal government play an evaluative role for STI/SS's? (WP) 

1-18. Enhancing diffusion of innovation through STI/SS's. (JG) 

1-19. Risk-taking in STI innovation. (DS) (See also Issue #3b.) 

Q-37. To what extent is market disaggregation *a barrier to STl/SS innovation, and what 

mechanisms could overcome this condition? 
Q-38. How does the rate of change of the technological (interest) profile of users affect STI/SS's? 

1-20. Organization and management of user groups to enhance STI use. (JG) (See also Issue ^2.) 
Q-27. What are (the) ways to motivate individual users to use STI/SS's? 

1-21. Design of STI/SS's to serve groups sized 100 to 1,000. (JG) 

1-22. Design, management, and function of intermediaries. (JG) 

1-23. Design and function of tertiary STI/SS's. (JG) (See also Issues #30, 31.) 

Q-19. What kinds of tertiary services are needed by different market segments?' 
Q-20. How can tertiary services be created utilizing existing institutions to optimize return on 
investment? 

Q-43. Have any designed STI systems accomplished what they intended? 

1-24. • (Nature, importance of) the STI reward structure. (JM) 

Q-25. What kinds of reward structures could be used to influence the use of STI by users? 

1-25. Improving feedback. (JKt) 

1-26. Methodology of conducting user studies by designers of STI/SS's. 

1-27. Getting the designers of STI/SS's to use the results of user studies. (AR) 
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1-28. User security. (CS) 

Q-14. What are the variety and effectiveness of user security measures? (JF, CS) 

1-29. Data base subsetting in social problem areas. (CS, JF) ^ 

Q-15. What strategies are employed by the developers and users of STI/SS's in extracting infor- 
mation relevant to emergent social problem areas (e.g., energy, environment)? (JF) 

1-30. Design and function of secondary services. (See Issues #23, 31.) 

Q-21. What kinds of secondary services are needed by different market segments? 

Q-22. How can these secondary services be created utilizing existing institutions to optimize return 
on investment? ** 

Q-43. Have any designed STI systems accomplished what they intended? 

1-31. Design and function of primary services. (See Issues #23, 30.) 

Q-23. What kinds of primary services are needed by different market segments? 

Q-24. How can these primary services be created utilizing existing institutions to optimize return 
on investment? 

Q-43. Have any designed STI systems accomplished what they intended? 

• It * 

1-32. STI/SS user motivation. 

Q-26. What are possible "personal retrieval tools" which could be used in STI/SS and how effective 
are they? 

Q-27. What are (the) ways to motivate individual users to use STl/SS's? 
Q-28. How can individual users be motivated to explore new fields of. STI? 

Q-44. What are the cognitive and motivational characteristics of different segments and what kinds 
and transforms of information will communicate to them? 

Q-45. What are the cognitive and motivational differences between people who select different 
professions and roles In science and technology and (their use of) STI? ^ 

Q-46. How can ^fleople with a record of successful STI innovations be detected, encouraged and 
challenged? 

Q-47. To what extent and in what ways can artistic mind styles supplement scientific and engi- 
neering mind styles in facilitating generation and communication of STI? 

1-33. STI/SS marketing. 

Q-29. What are (the information) purchasir^ behavior patterns of STI users? 

Q-30. What successful marketing technique! might be applicable in STI/SS marketing? 

Q-31. What new marketing techniques migh^ be applicable in STI/SS marketing? 

Q-47. To what extent and in what ways \ artistic mind styles supplement scientific and engi- 
neering mind styles In facilitating generilJon and communication of STI? 

Q'48. If a research field is dominated by ^ or a few companies, are there fewer kinds of STI 
systems than where there is competition? 
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1-34. Design of STl/SS's. 

Q-32. What needs to be done with information sources to improve efforts of suppliers to design 

STl/SS's more related to user behavior? 
Q-36. What ?.re existing industrial support practices for STI? 

Q-37. To what extent is market disaggregation a barrier to STI/SS innovation, and what 

mechanisms could overcome this condition? 
Q-43. Have any designed STI systems accomplished what they intended? 

Q-47. To what extent and in what ways can artistic mind styles supplement scientific and 
engineering mind styles in facilitating generation and communication of STI? 

Q-49. Could a computer be programmed to store high-frequency citations often used in one's 
papers and to change them according to the different styles of different journals? 

1-35. STI/SS management techniques. 

Q-33. What kinds of long-range planning techniques are needed by managers of STl/SS's, and how can 
they be acquired? 

Q-34. What is the relationship between organizational structure/climate and individual STI 
behavior and use? 

Q-35. How can organizational policies/structure/climate/etc. be changed to promote the use of 
STI by individuals? 

Q-46. How can people with a record of successful STI innovations be detected, encouraged and 
challenged? 

Q-47. To what extent and in what ways can artistic mind styles supplement scientific and 
engineering mind styles in facilitating generation and communication of STI? 

Q-50. Will journal problems change as the decrease in population growth diminishes the size of 
higher education and lowers the pressure to publish? 

Q-51. Could a national referral service supersede professional society directories? 

Q-52. How can invisible colleges be tapped for wisdom otherwise not evoked? 

Q-53. How can grapevines be tapped for state-of-the-art surveys? 

Q-54. Will a Journal of Winning Proposals be accepted? 

Q-55. How can all early stages of science be tapped in advance of formal publication for early 
warning systems as to new concepts, new approaches, new data, new interpretations and 
promising newcomers? 

1-36. Other forms of indicia should be tested, besides condensations (as in absracts) and subject-matter 
(as in key words); other approaches include level, approach, kind of mechanism discovered, etc. 

1-37. Ilv^y article should be tagged with a terse statement of its major claim, perhaps as its title. (See 
also Issue #47.) 

Q-56. Can NSF-sponsored Councils of Journal Editors in each field require authors to provide the 
main clairii? 

Q-57. Can appended "major claim" statements be made a requirement of publication? 
O-60. Who should tag papers with the major claim statements? 

1-38. Output measures are needed. 

1-39. Professionals in the field should periodically make predictions. 
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1-40. Since the state of STI is a function of the state of current technology, OSIS should support 
experiments on large numbers of permutations of current components. 

1-41. OSIS should encourage entreprenuers who couldn't afford tria! innovations otherwise. 

1-42. Other kinds and potential kinds of intermediaries should be recognized besides the traditional 
kinds. (See Issue #22.) 

1-43. The change in generations requires longitudinal and historical studies of STI; a new demography is 
needed of how science is done in the USA. 
Q-58. Do only burgeoning fields in ferment have STI problems? 
Q-59. Of what special sort are their problems (e.g., recruiting)? 

1-44. "Fugitive" files should be more accessible. ^ 
Q-59. What set of principles should guide development of "fugitive" files? 

1-45. The social matrix of STI— directories, phone numbers of knowledgeable people, etc.— is 
important. 

1-46. Technologists (who seek information on devices embodying solutions to problems) differ in 
information needs from scientists. 

1-47. There are no STI problems in three fields: law, patents and mathematics. This is primarily because • 
(a) precedent is important; (b) there are very precise f.orms and styles; (c) access tools are rigidly 
structured and well known; and (d) new developments are made explicit (i.e., explicit major 
'claims are required). (See also Issue #37.) 

Q-56. Can NSF-sponsored Councils of Journal Editors in each field require authors to provide the 
main claim? 

Q-57. Can appended "major claim" statements be made a requirement of pijblication? 
Q-60. Who should tag papers with the major claim statements? 

1-48. Tapping into privileged grapevines. 

Q-61. How can equity be obtained in STI services for those with STI problems? 
Q-62. How can all get access to information on privileged grapevines? 

1-49. User studies should be done not for the NSF but for client agencies who should retain STI 
researchers to serve them. 

1-50. On nonpatentable but desireable innovations, profit organizations should be aided by NSF to 
cost-share. 
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KEY STI ISSUES AND POTENTIALLY 
RESEARCHABLE QUESTIONS: 
Form Used in Evaluation Round Two 

Introduction. More than 200 scientific and technical infornnation (STI) issues and questions were 
identified by persons contributing ideas to the workshop. For analytical convenience, most of the issues 
and questions are grouped Into four functional levels depicted in the following figure. The four 
functions include STI system and service (a) design and redesign; (b) management; (c) operations; and 
(d) 'marketing. An additional level, (e) context, is used to pinpoint issues and questions normally outside 
the control of STI system and service managers. 

FOUR FUNCTIONS WITHIN THE CONTEXT OF SCIENTIFIC 
AND TECHNICAL INFORMATION SYSTEMS AND SERVICES (STI/SS'S)* 




*Not an attempt at a rigorous taxonomy or model of the STI domain, but merely a convenient 
way of grouping the STI issues and research questions. 



Rating. Please rate the importance of each issue (1) and the research ability of each question (Q) on 
a scale from 10 (highest) to 0 (lowest). Note that in most cases initials of contributors are included in 
parentheses after each issue and question. 



35 



APPENDIX B 



YOUR RATIIMG 
10 0 

(Highest) (Lowest) LEVEL A. STI/SS DESIGN AND REDESIGN 

issue Question 
(Importance) (Researchability) 

IA1. Design of STt/SS's. 

QA1. What needs to be done with information sources to 
improve efforts of suppliers to design STl/SS's more 
related to user behavior? * 
QA2. What kinds of STl/SS's (e.g., primary-secondary; hori- 
zontal-vertical) are needed by different marl<et 
segments? 

QA3. How can new STl/SS's be created utilizing existing 

institutions to optimize return on investment? 

QA4. How can existing STl/SS's (e.g., periodicals, computer- 
ized data bases, libraries) be modified to meet emer- 
gent information needs in major social problem areas 
(e.g., energy)? (JF) 

QA5. To what extent have existing STl/SS's accomplished 

their objectives? (MR) 
QA6. What STI/SS design alternatives exist for meeting needs 

of different sized groups (e.g., 100-1,000)? (JG) 
QA7. To what extent and in what ways can artistic mind 

styles supplement scientific and engineering mind styles 

in facilitating generation and communication of 

STI? (TM) 

QA8. Could a computer be programmed to store high- 
frequency citations often used in one's papers and to 
change them according to the different styles of 
different journals? (CT) 

QA9. How can existing STl/SS's be redesigned to become 

the "knowledge workshops" Doug Englebart at SRI 
talks about? (PA) 

QA10. Why has research in artificial intelligence, automata, 

and problem-solving been largely Ignored in the devel- 
opment of large-scale, integrative data bases (both 
primary and secondary)? What has prohibited transfer 
of this theory in mathematics to practice in designing 
STl/SS's? (JMa) 

QA11. Which journal problems will change as the decrease in 

population growth diminishes the size of higher educa- 
tion and lowers the pressure to publish? (MR) 
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YOUR RATING 
10 0 

(Highest) (Lowest) LEVEL A. STI/SS DESIGN AND REDESIGN (Continued) 

Issue Question 
(Importance) (Researchability) 

QA12. To what extent could a national referral "service 

supplement professional society directories and expe- 
dite communication between researchers and publics 
other than peers? (CT, PA) 

IA2. Risk-taking in STI/SS innovation. (DS) 

QA13. To what extent is market disaggregation a barrier to 

STI/SS innovation, and what mechanisms could over- 
come this condition? (DS) 

QA14. How does the rate of change of the technological 

interest profiles of users affect STI/SS's? 

QA15. What are existing industrial support practices for STI? 

QA16. How can people with a record of successful STI 

innnovations be detected, encouraged and chal- 
lenged? (TM) 

QA17. What can be done to encourage STi/SS entreprenuers 

who cannot otherwise afford trial Innovations? (PU) 

QA18. How can totally new STI/SS's and content evaluations 

be introduced into the STI sector? (TM) 

QA19. Will a Journal of Winning Proposals be accepted? (TM) 

I A3. Use of information concerning users in the design of 
STI/SS's. (CT) 

IA4. Getting designers of STI/SS's to use the results of user 
studies. (AR) 

QA20. How much account do designers of STI/SS's take of 

f actual user attitudes and behavior in designing their 

systems and services? (AR) 
QA21. How do STI/SS designers use the results of user 

studies? (AR) 

QA22. What changes in STI/SS design result from direct and 

indirect user feedback (e.g., through complaints initi- 
ated by users or systematic user studies)? (AR) 

. QA23. What problems (attitudinal and behavioral) trouble 

STI/SS designers and managers? (AR) 

QA24. How do STI/SS designers and managers evaluate the 

effectiveness of operating or planned systems and 
services? (AR) 

QA25. What cost/benefit ratios do managers of R&D and 

other users employ to evaluate or plan the size and/or 
complexity of STI/SS's? (AR) 
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APPENDIX B (Continued) 



YOUR RATING 

10 0 

(Highest) (Lowest) 
Issue Question 
(Importance) (Researchability) 



LEVEL A. STI/SS DESIGN AND REDESIGN (Concluded) 

IA5. Availability of proprietary user behavior information to STI 
academic researchers. (AR) 

IA6. Choice of target areas for user studies. (TG) 

IA7. Other kinds and potential kinds of intermediaries besides the 
traditional kinds. (IVIR) 

IA8. Integration, standardization, and tailoring of STI data banks 
(vertical vs. horizontal) for internal use in industry. (CS) 
QA26. What alternatives exist for standardizing the structure 
of STI/SS's? (JF) 

IA9. Relevance of STI to major social problem areas (e.g., energy, 
environment). (JF) 

QA27. How is STI integrated with other classes of data and 
information (e.g., legal, social) in the development and 
implementation of public policy? (JF, JIVIa) 

QA28. How can STI generators be influenced to become more 
participative in responding to policy use requirements 
for STI? (JF) 

QA29. To what extent has STI from one field been integrated 
with STI from other fields? Where and under what 
conditions have subsets of STI across fields been 
constructed? (JlVla) 

QA30. What transformations in STI occur as innovations move 
along the research-development-commercialization con- 
tinuum? (JF) 
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YOUR RATING 

10 0 

(Highest) (Lowest) 
Issue ^ Question 
(Importance) (Researchability) 



LEVELS. STI/SS MANAGEMENT 



IB1 . Application of economic analysis techniques to the ST I 
sector. (PA, WC) 

QB1. What kinds of economic analysis are potentially 
appropriate to STI? What kinds of economic analysis 
are most useful for STI? (CT) 

IB2. Economic modeling of scientific- and technical communi- 
cation. (PA, CH) 

QB2. How can the STI sector be modeled effectively for 

analysis and understanding by '^economists and STI/SS 

managers? (JG, TM, PA) 
QB3. How can the value of information be measured at 

various levels? (DS) 
QB4. What economic factors affect the impact of 

STI/SS's? (AC) 

QB5. Do the economic forces of supply and demand operate 
at the level of an international market? If so, are the 
patterns similar to the mechanisms of mircoeconomic 
analysis? Are similar measurement techniques appli- 
cable? (JH) 

IBS. STI/SS management techniques. 

QB6. What kinds of long-range planning techniques are 
needed by managers of STI/SS's, and how can they be 
acquired? (PU) 

QB7. What is the relationship between organizational struc- 
ture/climate and individual STI behavior and 
use? (AC) 

QB8. How can organizational policies/structure/climate/etc. 
be changed to promote the use of STI by indivi- 
duals? (AC) 

IB4. Financing STI/SS's. (JG) 

QB9. What are the pricing policies of various STI/SS own- 
ers? (PU) 

QB10. What are the costs and benefits of various STI/ 
SS's? (AR) 

QB11. How can the efficiency of STI/SS's be improved? (JF, 
DS) 

QB12. What are the financial incentives for influencing the 
STI/SS structure? (JG) 
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YOUR RATING 

10 0 

(Highest) (Lowest) 
Issue Question 
(Importance) (Researchability) 



LEVEL B. STI/SS MANAGEMENT (CONTINUED) 



QB13. What will new STI , handling methods cost compared to 
the old? That is, are there cost savings or increased 
costs? (WC) 

QB14. How do capital requirements and profit potential 
affect the growth of STI/SS's? (AC) 

QB15. What approaches are taken to interservice pricing? 
What are the consequences of different interservice 
pricing options for participating STI/SS's? (JG) 



IBB. The management of intermediaries. (JG, PU) 

QB16. How can STI relayers and service agents best perform 

their activities? (NL) 
QB17. How are reviewers selected? Is their selection more. 

geared to the needs of the particular disciplines or 

problem areas, or do politics play an overriding 

roie? (JMa) 



IB6. Use of information concerning users in the management of 
STI/SS's. (CT) 

QB18. What methods would be appropriate for permitting 
users of user studies to feed into and evaluate such 
studies? (TG) 



IB7. International cooperation in the development and management 
of STlAS^jS's. (JH) 

QB19. What international policies and programs relate to the 
operation of STI/SS's? (JH) 

QB20. To what extent do such policies and programs facili- 
tate or inhibit international information exchange 
among more prosperous and less prosperous 
nations? (JH) 

QB21. To what extent do political processes reinforce and 
perpetuate the concentration of STI/SS re- 
sources? (JH) 



IBS, Reduction in journal subscriptions. (AR) 

QB22. To what extent are different STI users cutting the 

number of their journal subscriptions? (AR) 
QB23. What factors (e.g., rising subscription charges) are 

associated with cuts in journal subscriptions and what 

alternatives, if any, can be developed to counteract 

such cuts? (JF) 
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YOUR RATING 
10 0 

(Highest) (Lowest) LEVEL B. STI/SS MANAGEMENT (Concluded) 
Issue Question 
(Importance) ( Researchability) 
jB9. Evaluation of STI/SS impacts. (MR) 

QB24. What should be the government's (e.g., federal, state) 

./esponsibility in evaluating STI/SS's? (WP) 

._ QB25. How can the "current awareness" effect of STI/SS's be 

measured? (MR) 

QB26. What will be the change in productivity of individuals 

served by introducing new information/data handling 
methods? (WC) 

QB27. What will be the perceived changes in the overall 

performance (e.g., profits) of the organizations within 
which new STI/SS's are embedded? (WC) 

QB28. What factors affect the stability and continuity of 

STI/SS's? (CH) 

QB29. What affect does the introduction of new STI/SS's 

have on the use of existing STI/SS's (e.g., increased or 
decreased use)? (JF) 
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YOUR RATING 

10 0 

(Highest) (Lowe st^ [ 

Issue QuestiCii 
(Importance) (Researchabllity) 



LEVEL C. STI/SS OPERATIONS 



IC1. Editorial practice. (JF) 

.QCI. Can NSF-sponsored Councils of Journal Editors In each 
field require authors to provide the main claim? (TM) 

QC2. Can appended "major claim" statements be made a 
\ requirement of publication? Who should tag papers 
with the major claim statements? (WP, TIVI) 

QC3. How useful is the abstract as a tool for assessing' the 
potential relevance of an article? What alternatives to 
the abstract (e.g., using only the first or last paragraph) 
exist and what would be the technical and financial 
impacts of employing them? (JMa) 

QC4. What other forms of indicia should be tested, besides 
condensations (as in abstracts) and subject-matter (as 
in key words)? Other approaches might include level, 
approach, kind of mechanism discovered, etc. (TM) 

QC5. To what extent does the use of professional jargon, 
acronyms, and complex Supporting mathematical form- 
ulae inhibit the readability and use of material by users 
not in a profession or the non-specialist within a 
profession? How would elimination of such language 
hamper the specialist? (JMa) 

IC2. Content quality control. (DS) 

QC6. What quality control mechanisms are feasible? (JG) 
QC7. What kind of quality control mechanisms are desir- 
able? (JG) 

QC8. How can the value of information per se be measured 
at various levels of usage? (DS) 

IC3. Inclusion of proprietary information/data in STI/SS's. (DS) 

IC4. Data base subsetting in social problem areas. (CS, JF) 

QC9. What strategies are employed by STI/SS users in 
extracting information relevant to emergent social 
.. problem areas (e.g., energy, environment)? (JF) 

IC5. STI/SS networking. (PA) 

IC6. Centralization versus decentralization in STI delivery. (TM, 
PU) 

IC7. STI packaging. (JF) 



42 



YOUR RATING 

10 0 

(Highest) (Lowest) 
Issue Question 
(Importance) ((Researchabllity) 



- APPENDIX B (Continued) 



LEVEL C. STI/SS OPERATIONS (Concluded) 

QC10. What is the feasibility of breaking down* the documents 
in a publication service into parts that can be used to 
compose new information pacRages? (JF) 

QC11. What are the cost/effectiveness ratios of different 
ways of packaging STI? In particular, what are the 
most .useful and least useful packaging practices for 
presenting the results of computerized literature search 
services? (JF) 

ICS. Restrains on information flow. (JMa) 

QC12. How do different STI media restrict the transmission 
of information to potential end users? How amenable 
are those restrictions to modification? (JMa) 

QC13. In what ways do workshops/seminars/conferences fa- 
cilitate and inhibit scientific and technical com- 
munication? (JMa) 

IC9. Industrial user security. (OS) 

QC14. What are the variety and effectiveness of user security 
measures? (JF, CS) 
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YOUR RATING 
10 0 

(Highest) ■ (Lowest) LEVEL D. STI/SS MARKETING 

Issue Question 
(Importance) (Researchability) 

- ID1. STI/SS marketing. (JF) 

QD1. To what extent have STI/SS managers used market 
research techniques to pinpoint potential user groups 
and to develop promotional appeals? (JF) 

QD2. What incentives and responsibilities motivate STI/SS 

managers to make their SS's more useful? What 
incentives might be offered that would motivate 
STI/SS managers, in their own interest, to increase 
their market research activity? (JF) 

QD3. What successful marketing techniques might be appli- 

-7.^ cable in STI/SS marketing? What new marketing 

techniques might be applicable in STI/SS marketing? 

QD4. How have market research efforts varied across differ- 
ent classes of STI/SS's? (JF) 

QD5. If a research field is dominated by one or a few 

companies, are there fewer kinds of STl systems than 
where there is competition? (CT) 

ID2. Market segmentation. (JG, JF) 

QD6. Who are the user groups and what are the bases for 

categorizing them? (TS, MR, JF) 

QD7. What different ways can "market segmentation" for 

STl users be done? (JG) 

\ QD8. What kinds of primary, secondary, and tertiary 

STi/SS's are needed by different market seg- 
ments? (DS) 

QD9. What are the information and data purchasing behavior 

patterns of STI/SS users? (JF) 

QD10. Within segments, what are the individual STl behavior 

(e.g., acquisition, use) patterns? What are the con- 
straints? What systems meet those contralnts? (MR) 

QDII.What are the critical variables that cause variance in 

information requriements and uses? (PA) 

QD12. How much time do different users invest in using 

various STI/SS's? (CH) 

,QD13. How much are users, as individuals or through their 

organizations, willing to pay for STI/SS's? (CH) 
• QD14. How do technology consumers and application con- 

sumers get their hands on STl and data they need 
when they want it and in a form they can use? (WC) 
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YOUR RATING ^ 
10 0 

(Highest) (Lowest) LEVEL D. STI/SS MARKETING (Continued) 

Issue Questions 
(Importance) (Researchability) 

QD15. What are the cognitive and motivational characteristics 

of different segments and what kinds and transforms 
of information will communicate to them? (TIVI) 

ID3. STI systems and services for public interest groups. (JKg) 

QD16. How will public interest groups respond to various 

types of STI/SS's and STI interpretations? (JKg, AR) 
QD17. How will public interest groups stimulate various types 

of STI/SS's and STI interpretations? (JKg, AR) 
QD18. How do public interest groups acquire STI/SS's? (JKg) 

ID4. STI/SS user training. (JF) 

QD19. How are scientists, engineers, managers, and others 

trained to use STI/SS's? (JF) 
QD20. What kind of training is needed to prepare scientists, 

engineers, and other STI users for interesting and 

successful use of STI/SS's? (AR) 
QD21. How much do alternative approaches to STI/SS user 

training cost, who pays, and what are the results? (JF) 

QD22. How can current approaches to STI/SS user education 

be improved? (JF) 

ID5. Nature of and leverage points in the STI reward struc- 
ture. (JM) 

QD23. What kinds of reward structures could be used to 

influence the use of STI by users? 

QD24. What factors influence end users' decisions to employ 

different STI/SS's? (JF) 

QD25. What barriers to STI/SS access or use do different STI 

users encounter? How amenable are those barriers to 
control? (JF) 

QD26. How can individual users be motivated to use existing 

STI/SS's and to explore the possibility of using new 
STI/SS's? (AR) 

QD27. What personal retrieval tools do STI users employ in 

accessing STI/SS's and how effective are they? 

■ ID6. User role in influencing the management of STI/SS's. (CT, 

AR) 

QD28. How can users be integrated into the management of 

STI/SS's? (JMa) 
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YOUR RATING i 
10 0 

(Highest) (Lowest) LEVEL D. STI/SS MARKETING (Concluded) 

Issue Question 
(Importance) (Researchability) 

ID7. Access to personal files. (WP) 

— QD 29. What principles should guide the development of 

personal files? (CT) 

, IDS. Access to privileged grapevines. (TM) 

QD30. How can equity be obtained in the STI services for 

those with STI problems? 

QD31. How can all get access to information on privileged 

grapevines? 

QP32. How can grapevines be tapped for state-of-the-art 

surveys? (CT) 

QD33. How can all early stages of science be tapped in 

advance of formal publication for early warning sys- 
tems as to new concepts, new approaches, new data, 
new interpretations and promising newcomers? (TM) 

QD34. How can invisible colleges be tapped for wisdom 

otherwise not evoked? (TM) 
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YOUR RATING 

10 0 

(Highest) (Lowest) ' LEVEL E. STI/SS CONTEXT 

Issue Question 
(Importance) (Researchability) 

IE1. The change in generations requires longitudinal and historical 

studies of STI; a new demography of how science is done is 
needed. (MR, WP) 

QE1. Do only burgeoning fields in ferment have STI prob- 
lems? (CT) 

QE2. What special problems (e.g., recruiting) exist in such 

fields? (TM) 

IE2. Enhancing the diffusion of innovation through STI/SS's. (JG) 

IE3. Impact of copyright law and practice on STI/SS's. (PU) 

QE3. What are (will be) the impacts of changes of copyright 

law or practice on STI/SS's? (DS, PU) 
QE4. What mechanisms can be used for handling the 

economic problems arising from current copyright 

practice? (PU) 

IE4. Impact of computer technology (e.g., ARPA, SRI Knowledge 

Center) on the STI sector. (PA) 

. QE5. To what extent is the critical mass of STI under 

^ control In existing information systems available in a '"^ 

computer-based, interactive retrieval environment? 
(PA) • 

IE5. Impact of new (e.g., nonlaboratory) modes of R&D and 

innovation on STI/SS's. (AR) 

* IE6. Impact of the trend toward finding and borrowing, rather than 

generating, STI on the STI sector. (DP) 

IE7. Impact of changes in academic research on STI generation and 

distribution. (MR) 

IE8. Impact of the trend toward cooperation, rather than competi- 
tion, on the structure and functioning of STI/SS's. (TM) 

IE9. Potential impact of newcomers to the STI field on the 

field. (TM) 

IE10. Social matrix of the STI/SS sector. (MR) 



1E11. Organization and management of user groups to enhance STI 
use. (JG) 



ERLC 
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YOUR RATING 

10 0 

(Highest) (Lowest) 



Issue 
(Importance) 



Question 
(Researchability) 



LEVEL E. STI/SS CONTEXT (Concluded) 



QE6. 



QE7. 



What organizational factors influence members to make 
use of STI/SS's? (AC) 

What are the cognitive and motivational differences 
between people who select different professions and 
roles in science and technology and STI? (MR) 



IE12. Duplication of research and development effort. (PA) 

QE8. To what extent is duplication of work occurring in 
areas of comparable research and development activity? 
(PA) 
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APPENDIX C 

1,000 POINT ALLOCATION RATING FORM 
AND QUANTATIVE RESULTS 

Introduction. Workshop participants assessed the relative importance of 47 STI issues in the 
second rating round, using a scale of 10 (highest importance) to 0 (lowest importance). For analytical 
convenience, the issues were grouped into four functional categories or levels: STI system and service (a) 
design and redesign, (b) management, (c) operations, and (d) marketing. An additional level, (e) context, 
was used to include issues normally beyond the control of STI system and service managers. 

Rating. Please use the following four-step procedure in this third, and final, rating of STI issues. 

• First, review the 30 issues selected. You'll note that they are the ones that received 
• the highest importance ratings in the second rating exercise. 

• Second, decide in your own mind how progress in this field might best be served by 
selecting a subset of the issues you think should be researched during the next three 
years. The key question you must address is just how much research effort should 
be invested in each of these issues between now and 1977. 

• Third, assume that 1,000 points represent the total resources available for R&D on 
these issues during the next three years. 

•\Fourth, distribute 1,000 points in any proportion you deem appropriate across the 
30 issues. Please do not let the total number of points you assign exceed 1,000. 

Comments. (1) Issues are ranked under each level from highest to lowest according to 
the average ratings resulting from the second round evaluations. (2) Issues retain the same 
identifying numbers as the ones used in the second round. (3) Please refer to Exhibit III if you wish to 
reexamine the potenially researchable questions related to each issue.* (4) In most cases the initials of 
contributors are included in parentheses after each issue. (5) Please suggest any changes in language you 
may wish to make. 



•"Exhibit III" was the term used to refer to the form employed in the second round of evaluation. 
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YOUR RATING 
(Number of points; 
if none, so state) 

915* 



710 
1,095 

530 

845 

425 
4,520 



LEVEL A. STI/SS DESIGN AND REDESIGN 

IA3. Use of information concerning users in the design of 
STI/SS's. (CT) 

IA1. Design of STI/SS's. 

IA9. Relevance of STI to major social problem areas (e.g., energy, 
environment). (JF) 

IA4. Getting designers of STI/SS's to use the results of user 
studies. (AR) 

1A7. Other kinds and potential kinds of intermediaries besides the 
traditional kinds. (MR) 

IA2. Risk-taking in STI/SS innovation. (DS) 

TOTAL POINTS ASSIGNED TO THIS LEVEL 
* * 



YOUR RATING 
(Number of points; 
if none, so state) 

805 



865 
385 
385 
600 

545 
585 

4,170 



LEVELS. STI/SS MANAGEMENT ^ 

IBI . AtJplication of economic analysis techniques to the STI 
sector. (PA, WC) 

1B9. Evaluation of STi/SS impacts. (MR) 

IB3. STI/SS management techniques. 

IB4. Financing STI/SS's. (JG) 

IBS. Use of information concerning users in the management of 
STI/SS's. (CT) 

IB5. The management of intermediaries. (JG, PU) 

IB2. Economic modeling of scientific and technical communi- 
cation. (PA, CH) 

TOTAL POINTS ASSIGNED TO THIS LEVEL 



•Total number of points allocated to this issue by participants during the Third Round of Evaluation. 
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YOUR RATING 
(Number of points; 
if none, so state) 

665 

715 

930 

2,310 



YOUR RATING 
(Number of points; 
if none, so state) 

1,240 

1,280 

430 



565 
770 

4,285 



APPENDIX C (Continued) 

LEVEL C. STI/SS OPERATIONS 
IC7. ST! packaging. (JF) 
IC5. STI/SS networking. (PA) 

IC4. Data base subsetting in social problem areas. (CS, JF) 

TOTAL POINTS ASSIGNED TO THIS LEVEL 
****** 

LEVEL D. STI/SS MARKETING 

ID1. STI/SS marketing. (JF) 

ID2. Market segmentation. (JG, JF) 

ID5. Nature of and leverage points in the STI reward struc- 
ture. (JM) 

ID4. STI/SS user training. (JF) 

ID6. User role in influencing the management of STI/SS's. (CT, 
AR) 

TOTAL POINTS ASSIGNED TO THIS LEVEL 
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APPENDIX C (Concluded) 



YOUR RATING 
(Number of points; 
if none, so state) 

990 

580 

405 
500 

335 

295 

210 
290 

100 

3,705 



LEVEL E. STI/SS CONTEXT 

IE3. Impact of copyright law and practice on STI/SS. (PU) 

IE4. Impact of computer technology (e.g., ARPA, SRI Knowledge 
Center) on the STI sector. (PA) 

IE2. Enhancing the diffusion of innovation through STI/SS's. (JG) 

IE11. Organization and management of user groups to enhance STI 
use. (JG) 

IE5. Impact of new (e.g., nonlaboratory) modes of R&D innova- 
tion on STi/SS's. (AR) 

IE6. Impact of the trend toward finding and borrowing, rather 
-' 7 than generating, STI in the STI sector. (DP) 

IE12. Duplication of research and development effort. (PA) 

IE7. Impact of changes in academic research on STI generation and 
distribution. (MR) 

IE8. Impact of the trend toward cooperation, rather than competi- 
tion, on the structure and functioning of STI/SS's. (TM) 

TOTAL POINTS ASSIGNED TO THIS LEVEL 
***** 



PLEASE CALCULATE YOUR 
FINAL POINT TALLY 



LEVEL NUMBER OF POINTS 



A 


— Design 


4,520 


B 


Management 


4,170 


C 


— Operations 


2,310 


D 


— Marketing 


4,285 


E 


— Context 


3705 




TOTAL 


18,990 



ERIC 



54 

r)2 



